## Summary
- Add embedding batch-size metadata to model responses and tenant
overrides.
- Validate embedding dimensions and batch limits across provider
verification and embedding requests.
- Expand validation tests for defaults, limits, and missing metadata.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
`compileChildrenPattern` re-implemented the delimiter-list compile
inline with two divergences from the shared
`CompileDelimiterListPattern`:
- It never stripped backticks, so a backtick-wrapped
`children_delimiter` like `` `###` `` matched the **literal wrapped
token** rather than the inner `###`.
- It sorted by **byte length** instead of rune count (`sortSlice`), so
multi-byte delimiters could be ordered incorrectly and a longer
delimiter could fail to win over a shorter prefix.
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
## Summary
Port the DSL tokenizer's `important_kwd` splitting into the Go
`Tokenizer` component so the indexed keyword array is byte-compatible
with the Python DSL pipeline and with the keyword-extraction prompt
contract.
- **Problem:** The Go component split `keywords` on the full ASCII+CJK
delimiter set (`utility.SplitKeywords`, regex `[,，;；、\r\n]+`), while the
DSL baseline `rag/flow/tokenizer/tokenizer.py:153` uses
`keywords.split(",")`, and `rag/prompts/keyword_prompt.md` instructs the
LLM to delimit keywords by **ENGLISH COMMA**. For a dataflow canvas that
includes the Tokenizer component, this divergence made Go's indexed
`important_kwd` differ from the Python-DSL-built index (CJK
commas/semicolons were split in Go but kept whole in Python).
- **Fix:** Use `strings.Split(kw, ",")` at `tokenizer.go:701`,
preserving empty middle elements to match Python's `"a,,b".split(",") ==
["a","","b"]`. The indexing fallback layer
(`internal/ingestion/task/indexdoc/process.go`) already mirrors the
Python multi-delimiter fallback (`dataflow_service.py:322`), so only the
component layer diverged and only it is changed.
## Test plan
- `TestTokenizerComponent_ImportantKwd_CommaOnly` (no build tag, default
`go test ./...`): switches the tokenizer to the identity engine (no CGo
pool needed) and asserts `"kw1,kw2;kw3，kw4"` → `["kw1","kw2;kw3，kw4"]`;
also asserts `important_tks` still tokenizes the full keyword string.
- `TestTokenizerComponent_Invoke_KeywordSplitCommaOnly` (`integration`
tag, real CGo analyzer): covers comma-split, CJK/semicolon-not-split,
and empty-middle preservation.
- Both tiers pass (unit `ok`, integration `ok`).
## Regression notes
- Intentional behavior change for canvases that include the Tokenizer
component: keywords containing `;`/`、`/newlines now stay as one keyword
(matching Python DSL) instead of being split. Re-indexing existing
Go-built data will change the `important_kwd` set — expected parity
cost, documented in code comments and commit message.
- Canvases without a Tokenizer component are unaffected (they hit the
unchanged multi-delimiter fallback).
- Other fields (`important_tks`, `questions`, `summary`, `text`) are
untouched; the `utility` import was removed cleanly.
## Summary
- Remove `splitOversizedUnit`, `splitAtomByTokenBudget` and `atomRE`
from `internal/ingestion/component/chunker/token.go`.
- Delete `split_oversized_guard_test.go` (added by #17740), which
guarded the removed atom-split behaviour.
- Drop the now-unused `wordCount`/`charCount` helpers from
`token_strict_cap_test.go`.
- Add `TestMergeByTokenSize_OversizedUnitStaysWhole` to pin the #17799
contract invariant (over-budget unit stays whole, never atom-split) on
the **text path**. The JSON path is already covered by
`TestMergeByTokenSizeFromJSON_OversizedUnitStaysWhole`.
## Why
The production merge path (`mergeByTokenSize` /
`mergeByTokenSizeFromJSON`) keeps over-budget units whole and relies on
the embedding/rerank layer to truncate them, per the TokenChunker
contract (#17799: remove atom-split, no hard_cap). The deleted helpers
implemented the opposite behaviour and had **no production caller**, so
they contradicted the contract and misled readers into thinking
atom-split was active.
## Parser vs chunker layering
Python's `_split_oversized_unit` lives at the **parser layer**
(pre-split before `naive_merge`), not in the chunker. Go's parser
backends are currently skeletons, so there is no parser-side equivalent
yet; if added later it belongs in `internal/parser/parser/*`, not the
chunker.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...`
passes; the new text-path test passes and the orphaned atom-split tests
are gone.
## Changes
- 3 files changed, 32 insertions(+), 250 deletions(-)
## Summary
Moves the canvas-debug parser page-cap injection out of the `task`
orchestrator and into a **debug-agnostic** `pipeline` helper, so
`PipelineExecutor` keeps only the orchestration skeleton (resolving one
of the P1 review findings: the executor was overloaded with
DSL/parser-param assembly).
### Changes
- **`pipeline/parser_page_cap.go`** (new):
- `BuildParserPageCapOverride(parserConfig, dsl, docType, capPages int,
parserComponentName string, familyOf)` — injects the
`ParserConfig[cpnID][family]["pages"]` cap through the same
`override_params` channel production uses. The cap value and family
resolution are injected by the caller, so the function carries no debug
semantics and is reusable for any page-cap scenario.
- `ExtractParserCpnID(dsl, parserComponentName)` — shared Parser cpnID
discovery from (optionally enveloped) DSL.
- `UnwrapCanvasDSL(raw []byte)` — exported single source of truth for
stripping the `{"dsl": {...}}` canvas envelope.
- `pipeline` does **not** import `component` (no reverse dependency);
callers inject `component.ComponentNameParser` /
`component.ParserFileFamily`.
- **`task/pipeline_executor.go`**: removed `injectDebugPageCap` (the
`debugPageCapPages = 2` constant stays in the task package). The debug
branch now calls `pipeline.BuildParserPageCapOverride(...)`.
- **`task/pipeline_executor.go` `warnUnknownComponentParams`**: fixed a
production no-op bug — it passed the enveloped DSL straight to
`ExtractAllComponentParams`, which silently errored and disabled the
unknown-cpnID guard. It now unwraps the envelope first.
- **`task/debug_result_dsl.go`**: reuses `pipeline.UnwrapCanvasDSL`
instead of a third inline envelope-unwrap copy.
### Behavior
No external debug-preview behavior changes. The three original
invariants are preserved exactly:
1. explicit `pages` caps under `cpnID+family` are respected (not
overwritten),
2. an empty family (unknown docType) is a no-op,
3. the injected shape is `[]any{[]any{1, capPages}}` (the
`[]any`-of-`[]any` form `NormalizePDFPages` requires).
## Test plan
- New `pipeline/parser_page_cap_test.go`: `BuildParserPageCapOverride`
(inject / respect-existing / unknown-family no-op / no-Parser no-op),
`ExtractParserCpnID` (enveloped + raw), `UnwrapCanvasDSL`.
- `task/debug_test.go`: `TestInjectDebugPageCap` migrated to the new
helper; new
`TestWarnUnknownComponentParamsDetectsUnknownCPNFromEnvelope` captures
the warning via `zaptest/observer` to prove the envelope no-op bug is
fixed.
- Both `internal/ingestion/pipeline` and `internal/ingestion/task` pass
`build.sh --test` (unit tier).
## Notes
- `TOKEN_CHUNKER_HANDOFF.md` is an unrelated untracked file and was
deliberately **not** included in this PR.
## Background
Issue #17889 asks that, when merging adjacent segments, the chunker
first
checks each segment's type and only merges **text** segments —
**table**,
**image**, and any other non-text type must each remain a standalone
chunk
and must never be merged with a neighbouring segment.
## Why this PR closes#17889 (no Go code change required)
After tracing the Go TokenChunker, the requirement is **already
satisfied**
on the structured (JSON / chunks) path. The type-aware rule is enforced
at
three layers in `internal/ingestion/component/chunker/`:
- `common.go:138` `itemDocType` derives the type from `doc_type_kwd`
(`"table"` -> `"table"`, `"image"` -> `"image"`, anything else ->
`"text"`).
It does **not** depend on the `ck_type` field being populated, so the
type
survives even when only `doc_type_kwd` is set (e.g. upstream
Title/Group/Hierarchy chunks).
- `token.go:756` `chunkFromItem` emits a non-text item as a single
standalone
chunk before the merge loop ever runs.
- `token.go:1050` `mergeByTokenSizeFromJSON` forces any non-text chunk
standalone (`if ck.CKType != "text"`); and `token.go:991` starts a
*fresh*
text chunk after a non-text chunk, so text on either side of a
table/image is never merged across it.
The only path without type information is the raw markdown/text/html
string
path (`PayloadFormatMarkdown/Text/HTML`), where the input is by contract
an
untyped string and `applyChildrenDelim` hard-codes `CKType: "text"` so
merging is correct. There is no non-text segment to merge there, so this
is
out of #17889's scope (which is about the merge logic).
## Why the Python side is deferred
The Python `naive` parser path does not thread a `ck_type` through to
`merge_paragraphs` / `naive_merge` / `naive_merge_with_images`
(`rag/nlp/__init__.py`): its parsers emit flat `(text, pos)` sections
plus a
parallel `section_images` list, and the type-aware `_merge_cks` rule
(`rag/nlp/__init__.py:1749`) is only wired into the docx path.
Propagating
`ck_type` end-to-end across every Python parser is a large refactor, so
it is
intentionally **not** part of this PR. The Go engine is the active
ingestion
path, and it already honors the rule.
## This PR
Adds a regression-lock (characterization) test, not a fix:
- `TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone` feeds a
`[text, table, text, image, text]` structured payload and asserts it
produces exactly five standalone chunks in the order
`text, table, text, image, text` — proving tables/images stay standalone
and text on either side is not merged across them.
Verified green:
```
bash build.sh --test -run TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone ./internal/ingestion/component/chunker/...
--- PASS: TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone (0.07s)
```
## Related
- Issue #17889
- PR #17808 (chunking refactor, merged)
- Contract doc #17799
Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
The generic `buildRequestBody` in `internal/entity/models/base_model.go`
unconditionally forwarded `ChatConfig.MaxTokens` as `"max_tokens"` for
every OpenAI-compatible provider.
Providers that need a different token field already delete or override
it after the call (e.g. Xiaomi uses `max_completion_tokens`, Replicate
uses `max_new_tokens`). This change stops setting `max_tokens` in the
shared builder so it only forwards the parameters common across
providers, and each provider remains free to set its own token limit
field.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.
This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
Go's `TokenChunker` kept the captured delimiter glued to the preceding
segment on **both** the primary (`chunkFromItem`) and secondary
(`children_delimiters`) split paths, while Python's reference
`token_chunker` drops it via `_split_text_by_pattern`
(`token_chunker.py:79-93`, used by both `_build_json_chunks` and
`_split_chunk_docs_by_children`). The divergence leaked the delimiter
into every emitted chunk's `text`.
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
## Summary
Update `conf/all_models.json`: replace legacy `max_tokens` with
`content_length` + `max_output` for all 2,178 chat/vision models, with
values verified against official vendor documentation.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Verify and populate `content_length` (context window) and `max_output`
(max generation tokens) for all **478 chat/vision models** across **47
provider configs**
- Data sourced from **official API documentation** via 12 parallel
agents + targeted web verification
- Update Go test assertions to match verified values
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #17835 (merged). The OVER_CAP / UNDER_CAP merge strategy
was threaded through `mergeDecision` and `mergeByTokenSizeFromJSON` as
an inlined `allowBoundaryOverflow bool` derived from `!c.param.UnderCap`
at three call sites. This replaces that with a named
`schema.MergeStrategy` enum.
## Why
- The `!c.param.UnderCap` inversion was hand-written in three places, so
a future strategy addition could silently drift between the JSON path
(`invokeTextPayload` / `invokeJSONPayload`) and the text path
(`mergeByTokenSize`) — no compile error, and the existing tests don't
cover all three sites with both strategies.
- The strategy concept was never named; `allowBoundaryOverflow` (true =
OVER_CAP) is a double-negation of `UnderCap` and reads opaquely at the
5th positional argument.
## What changed
- Add `schema.MergeStrategy` (`MergeOverCap` / `MergeUnderCap`)
mirroring Python's `rag/nlp/__init__.py` `MergeStrategy`, so Go and
Python stay on the same vocabulary.
- Expose `TokenChunkerParam.MergeStrategy()` derived from the
wire-facing `UnderCap bool` (existing `"under_cap"` configs keep working
— no schema break).
- `mergeDecision` and `mergeByTokenSizeFromJSON` now take
`schema.MergeStrategy` instead of `allowBoundaryOverflow bool`; the
three call sites pass `c.param.MergeStrategy()` (no `!`).
- Tests updated to pass the enum; added a guard test for the `UnderCap`
-> `MergeStrategy` mapping and an end-to-end test for UNDER_CAP on the
JSON path.
No behavior change: default remains OVER_CAP, `under_cap=true` still
selects UNDER_CAP.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...
./internal/ingestion/component/schema/...` — all green, including
`TestMergeByTokenSizeFromJSON_UnderCapNoOverflow`,
`TestMergeByTokenSize_UnderCapNoOverflow`,
`TestInvokeJSONPayload_UnderCapEndToEnd`, and
`TestTokenChunkerParamMergeStrategy`.
## Related issues
- Relates to #17835 — wired UNDER_CAP as a tested merge-strategy seam
(merged)
- Relates to #17799 — contract doc for token-chunker cap/delimiter
alignment
- Relates to #17808 — related chunker alignment work
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Remove the Go Report Card badge from `internal/harness/README.md`
because the service has been retired and no longer provides a repository
grade.
This is a documentation-only change. Validated with `git diff --check`.
## Problem
During ingestion, `indexdoc.ProcessChunksForPipeline` stamped
`ck["kb_id"]`
on every chunk. This was both:
- **a dead write** — `elasticsearch.InsertChunks` unconditionally
overwrites
the value with `datasetID` (`chunk.go:211`), so the producer's value
never
reached the index;
- **the wrong shape** — it was emitted as `[]string`, while both engines
actually need a single string.
This is the `kb_id` slice of the ingestion -> engine schema leak tracked
in
#17371: ingestion was carrying index-physical schema knowledge it should
not
own.
## Fix
Make the search engines the single owner of `kb_id` at the write
boundary,
and stop ingestion from emitting it:
- **Elasticsearch** (`chunk.go:211`) already sets `docCopy["kb_id"] =
datasetID`
— unchanged.
- **Infinity** (`chunk.go`) `InsertChunks` now stamps
`insertChunks[i]["kb_id"] = datasetID` right after
`transformChunkFields`
(previously it only *read/normalized* the producer value, which forced
ingestion to supply it). Both engines are now consistent.
- `ProcessChunksForPipeline` no longer stamps `kb_id` and the now-leaky
`kbID` parameter is removed. The same removal is propagated to
`ProcessPipelineOutputForGolden` and the `compare_pipeline_golden` dev
tool
(its `-kb-id` flag is dropped).
The stored `kb_id` value is byte-for-byte unchanged: `datasetID` passed
to
`InsertChunks` is `taskCtx.Doc.KbID`, i.e. the same id that was
previously
set on the producer chunk.
## Verification
- `bash build.sh --test ./internal/ingestion/task/indexdoc/...
./internal/engine/infinity/...`
— both green.
- `internal/ingestion/task` has **two pre-existing** failures
(`TestPipelineExecutor_Run_RealCanvasDSL_UsesGeneralPipeline`,
`TestRunPipeline_RealPipelineOutput_ProducesIndexFields`) that assert
`inserted chunk count = 1, want 2` — a parser/assertion mismatch (the Go
parser merges the 2-paragraph fixture into 1 chunk). They are unrelated
to
this change, which never touches chunk counting. The `kb_id`-related
test
failure this change would otherwise introduce is fixed by updating the
tests
below.
- Updated the pinning unit test:
`TestProcessChunksForPipeline_SetsDocID`
(formerly `...SetsDocIDAndKBID`) now asserts `kb_id` is **not** set by
the
producer. Removed the `kb_id` assertion and the now-dead
`taskChunkFieldEqualsStr` helper from
`pipeline_real_integration_test.go`.
## Scope
This closes only the `kb_id` portion of #17371. The remaining
index-physical
fields (`docnm_kwd`, `create_timestamp_flt`, `page_num_int`/`top_int`/
`position_int`, etc.) are intentionally left for a follow-up (P2).
## Problem
`kg_test.go`'s `getTestConfig()` returned `map[string]interface{}`,
which no longer matches `NewEngine`'s `config.ElasticsearchConfig`
signature after the config package moved to `internal/server/config`.
This broke compilation of the `elasticsearch` package under `-tags
integration` (kg_test.go:38).
## Fix
Make `getTestConfig()` return the typed `config.ElasticsearchConfig` so
the integration tier builds again.
## Scope
Fixes test compilation only; no production code changed. Unrelated to PR
#17802 (kb_id single-string + T0 read-back baseline), so it is tracked
in its own PR to keep that refactor focused.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Relate to #17284.
## Summary
Batch 5/6 migrated the rest of the Go model drivers onto the shared HTTP
helpers (`doRequest`, `doStreamRequest`, `applyAuth`). This PR completes
the batch for the remaining OpenAI-compatible chat-streaming drivers
that were still hand-writing HTTP requests:
- **7 drop-in migrations**: deepseek, gpustack, groq, longcat, moonshot,
openai, siliconflow
- **1 adapter migration**: minimax (relocated its `io.Pipe`
error-interception into the `doStreamRequest` handler)
- **1 full migration**: azure_openai (all four paths: chat, streaming,
embeddings, list-models) plus the auth header hook
- **1 receiver fix**: nvidia `NewInstance` value → pointer
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
Relate to #17284.
## Problem
`novitaHandleStream` guarded usage recording with `if found &&
chatConfig != nil`. When a caller passes a nil `*ChatConfig` — common in
the service layer (`model_chat.go`, `chat_pipeline.go`) — the streamed
token usage is dropped entirely.
The shared `HandleStreamingResponse` only uses `chatConfig` to expose
`UsageResult` and records usage whenever the stream carries it. Novita's
bespoke handler diverged from every other OpenAI-compatible streaming
driver.
## Fix
Record usage whenever the stream carries a usage event, mirroring
`HandleStreamingResponse`. `applyStreamUsage` already handles a nil
`chatConfig` internally (it only writes `chatConfig.UsageResult` when
non-nil), so the extra guard was doing nothing but dropping usage.
## Test
`TestNovitaStreamRecordsUsageWithoutChatConfig`:
- nil `chatConfig` + usage event → stream completes without error (guard
removed safely)
- non-nil `chatConfig` + usage event → `UsageResult` populated with the
streamed tokens
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
