## Summary
Fixes embedding inputs being emptied and documents chunker
heading-detection parity.
- `component/tokenizer.go`: `truncateForEmbedding` now mirrors Python
`common/token_utils.py:183-185` `truncate(string, max_len)` (keep the
first `max_len` tokens). For an unconfigured embedder reporting
`maxTokens <= 0`, instead of mirroring Python's `truncate` (which
returns `""` for `max_len <= 0` and would make the embeddings API reject
the batch with "inputs cannot be empty"), Go **clamps the limit to a
safe default** (`defaultEmbeddingTokenLimit = 8192`) so truncation stays
active and never produces empty inputs. The previous code returned `""`
for `maxTokens <= 10`, which emptied non-empty chunks into the batch and
triggered the API error.
- A **10-token safety margin** is reserved (mirroring Python's embedding
path `rag/svr/task_executor.py` which uses `mdl.max_length - 10`) so Go
does not over-send tokens to hard-capped embedding APIs; it is only
applied when the limit `> 10` so small limits remain non-empty.
- The clamp is applied **centrally inside `truncateForEmbedding`**,
covering both Builtin and generic callers — no separate
`model_service.go` change is required.
- `component/chunker/title.go`: comment-only — documents the
already-ported PDF-outline detection (omission 1.5 / Gap C) and the
hardcoded `BULLET_PATTERN` fallback (omission 1.7 / Gap C), porting
`common.py:_outline_similarity` and `rag/nlp` `BULLET_PATTERN`.
- `tokenizer_unit_test.go`: update truncation expectations to the new
positive-keeps-tokens / non-positive-clamps-to-default behaviour.
## Test plan
- `tokenizer_unit_test.go` updated for the new truncation behaviour
(`TestTruncateForEmbedding_SmallMaxTokens`,
`TestTruncateForEmbedding_UnconfiguredClampsToDefault`).
- `./build.sh --test ./internal/ingestion/component/` passes.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Ports doc-level auto-metadata extraction to Go and adds the
knowledge_compiler component with scheduler/routing. Fixes Extractor
metadata injection type assertion and enable_metadata default-on.
## Summary
Remediates CVE-2026-0994 (HIGH) in `protobuf` by adding
`protobuf>=5.29.6` to `constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-0994 | HIGH | protobuf | 5.29.5 | 5.29.6 |
`protobuf` is a transitive dependency pulled in by `google-api-core`,
`googleapis-common-protos`, `grpcio-status`, and `opentelemetry-proto`.
It was stuck at 5.29.5 because the lockfile hadn't been re-resolved.
## Summary
- StepFun serves domestic (China) and international markets from
different domains.
- Add a `china` region URL (`api.stepfun.com`) alongside the existing
`default` (`api.stepfun.ai`) so tenants in China can route correctly.
## Change
- `conf/models/stepfun.json`: add `"china":
"https://api.stepfun.com/v1"` to the `url` map.
## Why
- Chinese AI/LLM providers commonly maintain separate URLs for domestic
vs. international markets. Keeping both options available matches the
existing pattern used by other providers (e.g. GiteeAI).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- Add provider-local Chat, Embedding, and Rerank response structures
with usage mapping.
- Align streaming usage and tool-call state handling across Gitee,
OpenRouter, Jiekou.AI, and Hunyuan.
- Preserve Jina's non-streaming behavior and explicit unsupported
streaming response, while fixing Jiekou.AI thinking=false handling.
## Summary
When `STORAGE_IMPL=AWS_S3`, the Admin Service status page keeps showing
MinIO. Three things conspire:
1. `admin/server/config.py::load_configurations` only knows the
`minio` and `minio_0` config keys. An `s3` block lands on the
`case _:` branch and logs `Unknown configuration key: s3`
(issue #17294).
2. `admin/server/services.py::ServiceMgr.get_all_services` filters
retrieval services by `DOC_ENGINE` but has no equivalent filter
for `file_store` services by `STORAGE_IMPL`. A stale MinIO block
is returned regardless of the active backend.
3. The wired health check is hardcoded to `check_minio_alive`, which
calls `settings.MINIO['host']`. With `STORAGE_IMPL=AWS_S3` that
block is uninitialized, so the check always times out.
Fixes#17294
## Summary
Remediates three HIGH severity CVEs in `pyasn1` by bumping the existing
`constraint-dependencies` entry from `>=0.6.3` to `>=0.6.4` in
`pyproject.toml`.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-59884 | HIGH | pyasn1 | 0.6.3 | 0.6.4 |
| CVE-2026-59885 | HIGH | pyasn1 | 0.6.3 | 0.6.4 |
| CVE-2026-59886 | HIGH | pyasn1 | 0.6.3 | 0.6.4 |
`pyasn1` is a transitive dependency (pulled in via `google-auth` / `rsa`
/ `pyasn1-modules`). A previous constraint (`>=0.6.3`) was added for
CVE-2026-30922 but two additional vulnerabilities were disclosed that
require 0.6.4.
### Summary
The **aimlapi.com** provider added in #17311 does not identify itself on
any of its outgoing requests, so its traffic cannot be attributed to the
integration. This PR adds the two headers AIMLAPI expects —
`X-AIMLAPI-Source` and `X-AIMLAPI-Partner-ID` — to every request the
provider makes.
### Summary
The Go Anthropic driver's `ChatStreamlyWithSender`
(internal/entity/models/anthropic.go) was a stub that always returned
`"no such method"`, so any caller requesting a streamed response from a
Claude model via the Go path failed outright — diverging from the Python
`AnthropicCV` driver, which already supports streaming.
This implements the method by opening the Messages API with
`stream=true` and parsing the SSE response via the shared
`ParseSSEStream` helper, forwarding `text_delta`/`thinking_delta`
content through the `sender` callback and treating `message_stop` as the
terminal event — consistent with the other Go drivers in this package
(e.g. Cohere).
Fixes#17333
---------
Co-authored-by: Abhay Yadav <abhayyadav@Abhays-MacBook-Air.local>
### Summary
Add token usage reporting for the LongCat (Meituan) model provider.
Related
to #17284.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
