…rphaned clusters
### Summary
**Problem**
When deleting a document from the knowledge base, the parent cluster
could never be located and was therefore never cleaned up, leaving
orphaned clusters in Infinity indefinitely.
**Root Cause**
InfinityConnection.field_keyword() treats all _kwd-suffixed columns as
keyword lists and applies .split("###") to them in get_fields().
parent_kwd was missing from the exclusion list. This caused a parent
cluster name like "Toronto Transit & Authority Control 5e8d5eb6" to
become ['Toronto Transit & Authority Control 5e8d5eb6'] (a
single-element list). When that list was fed into _nav_cluster_id(), it
produced a different hash than the actual cluster's hash (computed from
the plain string), so every deletion-time cluster lookup silently
failed.
**Fix**
Added "parent_kwd" to the exclusion list in field_keyword(), alongside
docnm_kwd, important_kwd, question_kwd, etc.
## Summary
Remediates nine litellm CVEs (3 CRITICAL, 6 HIGH) by upgrading the exact
pin from `==1.82.5` to `==1.84.0` in `pyproject.toml`.
| CVE | Severity | Fixed in |
|---|---|---|
| CVE-2026-35030 | CRITICAL | 1.83.0 |
| CVE-2026-42208 | CRITICAL | 1.83.7 |
| CVE-2026-49468 | CRITICAL | 1.84.0 |
| CVE-2026-35029 | HIGH | 1.83.0 |
| CVE-2026-40217 | HIGH | 1.83.10 |
| CVE-2026-42203 | HIGH | 1.83.7 |
| CVE-2026-42271 | HIGH | 1.83.7 |
| CVE-2026-47101 | HIGH | 1.83.14 |
| CVE-2026-47102 | HIGH | 1.83.10 |
`litellm` was pinned to `==1.82.5` because versions 1.82.6 - 1.82.8 had
a broken `litellm.caching.caching` import chain, and 1.88.0 had a broken
`litellm.integrations.custom_logger` import.
Both issues are resolved in 1.84.0.
The exact pin is retained (rather than a range) due to litellm's history
of introducing regressions on minor bumps.
## Problem
`TestMergeByTokenSizeFromJSON_ClampsOverlappedPct` reused a single
`items` fixture across four calls to `mergeByTokenSizeFromJSON`.
`mergeByTokenSizeFromJSON` mutates its `perItem` argument in place and
returns the same backing array (token.go: `perItem[idx] = merged`). As a
result:
- The 2nd and later calls merged already-merged chunks instead of the
original input.
- Because the returned slice aliases the input, later calls silently
overwrote the earlier results (`at100`, `at0`, ...).
- `reflect.DeepEqual(at100, at150)` was therefore vacuously true — the
test was a **false positive** that never actually exercised the clamp.
It would still pass even if the clamp were broken.
This is exactly the defect flagged in the review comment on PR #17396.
## Fix
Add a `clampOverlapFixture()` factory and build a fresh fixture for
every call, so that 150 / 1e300 / -5 / -1e300 are applied to the
original input.
## Verification
- The test passes after the fix.
- When the clamp logic was temporarily disabled, the test **failed**
(panic at token.go:768 — the negative index produced by an out-of-range
pct), proving the fixed test is no longer a false positive and can catch
a clamp regression.
## Scope
Test file only. No production code change (verified `token.go` is
identical to `upstream/main`).
Refs: review comment on PR #17396.
## Summary
Remediates five nltk CVEs by adding `nltk>=3.10.0` to
`constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Installed | Fixed in |
|---|---|---|---|
| CVE-2025-14009 | CRITICAL | 3.9.2 | 3.9.3 |
| CVE-2026-0846 | HIGH | 3.9.2 | 3.9.3 |
| CVE-2026-0847 | HIGH | 3.9.2 | — |
| CVE-2026-33231 | HIGH | 3.9.2 | 3.9.4 |
| CVE-2026-33236 | HIGH | 3.9.2 | — |
| CVE-2026-54293 | HIGH | 3.9.2 | 3.10.0 |
| CVE-2026-33230 | MEDIUM | 3.9.2 | 3.9.4 |
CVE-2026-0847 and CVE-2026-33236 have no confirmed fixed version.
`nltk` is a transitive dependency (pulled in by `crawl4ai` and
`infinity-sdk`). No package in the graph caps it below 3.10.0. It was
stuck at 3.9.2 simply because the lockfile had never been re-resolved.
## Summary
- Parse chat, embedding, and rerank usage from provider responses
- Record usage with the correct model type even when no usage sink is
provided
- Cover SiliconFlow, Aliyun, Huawei Cloud, Qiniu, and VolcEngine
response formats
## Summary
Closes two Chunker migration gaps documented in
`docs/migration_python_go_diff.md`
(diffs **2.6** and **1.7**), improving parity with the Python ingestion
pipeline.
This is the code portion of commit `261e1fd0b` on branch
`fix/batch-3-4`; the
migration document itself is tracked separately (untracked in this
commit).
### Diff 2.6 — `overlapped_percent` missing Python normalization
Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`)
accepts a
`[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it
by 100,
`int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a
raw
`[0,90]` percentage and **rejected** out-of-range input, so a Python
config
passing `0.1` (meaning 10%) silently produced ~0% overlap.
Added `normalizeOverlappedPercent`
(`internal/ingestion/component/chunker/common.go`) mirroring the Python
helper:
- parses numbers and numeric strings (mirrors Python `float()`; bad /
`NaN` / `Inf` → `0`),
- `0 < v < 1 → v *= 100`,
- `int()` truncation,
- clamps to `[0,90]`.
Wired into `tokenChunkerParam.Update` (`token.go`); the merge math
(`token.go:705`, `(100-x)/100`) already matched Python.
`TokenChunkerParam.Validate`
(`schema/chunker.go`) now only guards direct struct construction.
Tests: `TestNormalizeOverlappedPercent`, extended
`TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp
inputs),
new `TestTokenChunker_NormalizesOverlappedPercent`. The existing
reject-test cases
for `<0` / `>90` were removed because they are now normalized/clamped
(Python parity).
### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback
`resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when
outline +
regex + layout all yield body level, `bulletsCategory` selects the
best-matching
bullet-pattern group (Chinese legal / numbering / Chinese numbering /
English legal
— mirroring `rag/nlp/__init__.py:258-320`) and assigns structural
levels. Guarded by
`allBodyLevel` so it never overrides an existing outline/regex level.
Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests).
### Incidental test adjustments included in the commit
- `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to
reflect the
post-normalization 30% semantics.
- `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` →
`LoadFromIngestionTask(ctx, task)` for the new context-first signature.
## Verification
`bash build.sh --test ./internal/ingestion/component/...` — chunker +
schema suites
pass, no regression (CGO build).
## Migration doc reference
`docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked
**Fixed** for
these changes.
## Summary
Improve the UX of the **"Delimiter for text"** field on the dataset
configuration page. The field is a single string with a backtick-based
mini-syntax, but both the tooltip and the surrounding UI failed to
surface what delimiters the backend would actually derive from a given
value — leaving users to discover by trial-and-error that the same
string produces different splits depending on file type (see #7436,
#4704, #9680).
