## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Fixes#17202 (and complements #12109).
## Problem
`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.
A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.
Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.
## Fix
Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:
1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
```python
if cks[-1] == "":
cks[-1] = t; tk_nums[-1] = tnum; return
if tk_nums[-1] + tnum <= chunk_token_num:
cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
cks.append(t); tk_nums.append(tnum)
```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.
2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.
3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.
A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.
## Result on the dataset above
| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |
## Tests
- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.
All 22 unit tests pass on the host venv:
```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK
test_naive_merge.py::test_images_oversized_section_is_split OK
test_naive_merge.py::test_images_custom_delimiter_preserved OK
test_naive_merge.py::test_images_plain_string_input OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK
test_naive_merge.py::test_strict_cap_single_overlong_section_… OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK
test_txt_parser.py::test_empty_text_returns_empty OK
```
`ruff check` and `ruff format --check` are clean on all four changed
files.
## Out of scope
- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.
Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Closes#17384.
## Summary
Drops a dead `re.I` flag from two outlier delimiter-parsing sites and
adds regression tests so the inconsistency can't creep back.
## What's wrong
Two of the six delimiter-parsing implementations pass `re.I` to
`re.finditer`:
- `rag/nlp/__init__.py::get_delimiters` (line 1633)
- `deepdoc/parser/txt_parser.py::parser_txt` (line 51)
The other four implementations correctly omit `re.I`:
- `rag/nlp/__init__.py::naive_merge` custom-delimiter path (line 1195)
- `rag/nlp/__init__.py::naive_merge_with_images` custom-delimiter path
(line 1269)
- `rag/nlp/__init__.py::_build_cks` (line 1389)
- `rag/flow/chunker/token_chunker.py` (line 73)
## Why this matters (and why it doesn't break anything)
The flag is **dead code** today. Verified empirically with a Python
REPL:
```python
>>> import re
>>> for m in re.finditer(r"`([^`]+)`", "`end`", re.I):
... print(repr(m.group(1)))
'end' # plain string, no flag attached
>>> re.split("(a)", "Class A is a Sample")
['Cl', 'a', '', 's', ' A i', 's', ' a Sample']
# Case-sensitive: only lowercase 'a' splits. Uppercase 'A' is preserved.
```
`re.I` does not propagate from `re.finditer` to `m.group(1)` or to
downstream `re.split` / `re.match` calls (which all omit `re.I`). So the
actual splitting behavior has always been case-sensitive — removing the
flag is a **defensive cleanup**, not a behavioral fix.
So why bother?
1. **Consistency** — the two sites were the only outliers in a six-way
implementation cluster. The three sibling sites in `rag/nlp/__init__.py`
already omit `re.I`, which strongly suggests the flag was accidental.
2. **Future-proofing** — a refactor could easily propagate the flag to a
downstream `re.split` call where it *would* change behavior. The tests
added here pin the case-sensitive semantics so that regression fails
loudly.
3. **Reader clarity** — the flag is misleading. Anyone reading
`re.finditer(..., re.I)` reasonably assumes case-insensitive matching,
then has to trace all downstream calls to discover it's a no-op.
## Changes
- `rag/nlp/__init__.py` — drop `re.I` from `get_delimiters` (line 1633).
- `deepdoc/parser/txt_parser.py` — drop `re.I` from `parser_txt` (line
51).
- `test/unit_test/rag/test_delimiter_case_sensitive.py` — new test file
with:
- 4 behavioral tests on `get_delimiters` (pattern output + `re.split`
round-trip).
- 3 end-to-end tests through `naive_merge` (bare-char +
backtick-wrapped, both cases).
- 2 parametrized static checks that `re.I` / `re.IGNORECASE` is not
present at either of the two `re.finditer` sites.
## Testing
```
$ pytest test/unit_test/rag/test_delimiter_case_sensitive.py -v
============================= 9 passed in 0.19s ==============================
```
All tests pass on the patched code. Before the patch, the 2 static
checks fail with a clear assertion message (the 7 behavioral tests pass
either way, confirming `re.I` was dead code).
## Related
- #17384 — the issue this PR closes. Note the issue's reproduction code
(`re.split(..., flags=re.I)`) doesn't actually match what the production
code does — the production `re.split` calls all omit `re.I`, which is
why current behavior is already case-sensitive. The fix here is still
valuable as a defensive cleanup + test coverage, but it's not a
behavioral fix per se.
- #17383 — broader parser consolidation (six implementations → one). The
fix here is independent and small enough to land first.
- #17385 — sibling UX PR (tooltip + live preview). Files are disjoint
(`web/src/**` vs `rag/nlp/**` + `deepdoc/parser/**`), so no interaction.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
### What problem does this PR solve?
`common/misc_utils.once` set `executed=True` before invoking the wrapped
function, so an exception on the first call permanently disabled future
calls and returned the cached `None`.
This change marks `executed=True` only after a successful call, allowing
retries after transient failures while preserving once-only behavior
after success. It also adds regression tests for retry-after-exception
and thread-safe single execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
## Summary
When `STORAGE_IMPL=AWS_S3`, the Admin Service status page keeps showing
MinIO. Three things conspire:
1. `admin/server/config.py::load_configurations` only knows the
`minio` and `minio_0` config keys. An `s3` block lands on the
`case _:` branch and logs `Unknown configuration key: s3`
(issue #17294).
2. `admin/server/services.py::ServiceMgr.get_all_services` filters
retrieval services by `DOC_ENGINE` but has no equivalent filter
for `file_store` services by `STORAGE_IMPL`. A stale MinIO block
is returned regardless of the active backend.
3. The wired health check is hardcoded to `check_minio_alive`, which
calls `settings.MINIO['host']`. With `STORAGE_IMPL=AWS_S3` that
block is uninitialized, so the check always times out.
Fixes#17294
### Summary
The **aimlapi.com** provider added in #17311 does not identify itself on
any of its outgoing requests, so its traffic cannot be attributed to the
integration. This PR adds the two headers AIMLAPI expects —
`X-AIMLAPI-Source` and `X-AIMLAPI-Partner-ID` — to every request the
provider makes.
### Summary
Adds a `tenki` sandbox provider that runs each agent code execution in a
disposable Tenki (https://tenki.cloud) microVM (create → exec → destroy,
no volumes or snapshots).
Registration mirrors PR #15039, configure `api_key` and `project_id` in
Admin > Sandbox Settings.
Both runtimes are covered:
- Python: `agent/sandbox/providers/tenki.py` (structured results +
artifact collection).
- Go: `internal/agent/sandbox/tenki.go`, mirroring the e2b provider and
wired into the provider manager.
`tenki-sandbox` is an optional dependency (it requires `protobuf>=6.31`,
which differs from RAGFlow's pinned gRPC stack), lazily imported with a
clear error when missing; installation is documented in the sandbox
quickstart.
Unit tests cover execution, structured results, artifacts
(symlink/size/extension limits), non-zero exit, timeout, error mapping,
and idempotent destroy.
---------
Co-authored-by: yiming.wang <yiming.wang@luxor.com>
## Summary
GreenPT is a European AI provider with an OpenAI-compatible API,
optimized infrastructure, and datacenters powered by 100% renewable
energy.
This adds native GreenPT support across RAGFlow’s Go-first provider
system and its Python compatibility layer:
- discovers the current catalog from `GET /v1/models`
- features `glm-5.2` and `kimi-k2.7-code` for chat and coding
- supports `green-embedding` through `/v1/embeddings`
- supports `green-rerank` through `/v1/rerank`
- supports `green-s` and `green-s-pro` speech-to-text through
`/v1/listen`
- adds provider configuration, UI icon, and supported-provider
documentation
### What problem does this PR solve?
Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.
`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.
Closes#17056Closes#5782Closes#7075
**What it does**
1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.
**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.
**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.
## What changed
### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.
### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.
### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).
### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.
## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
### Summary
`agent/tools/google.py` indexes `search["organic_results"]` directly
after `GoogleSearch(params).get_dict()`. serpapi returns `{"error":
...}` **without** an `organic_results` key on realistic conditions — an
invalid API key, an exhausted plan/quota, or a query that matched
nothing. that raised `KeyError('organic_results')`, which the tool's
retry loop then surfaced to the model as the opaque `"Google error:
'organic_results'"` instead of the real reason.
this guards the missing key and raises serpapi's actual `error` message
(with a clear fallback) into the existing retry/`_ERROR` path, so the
model sees e.g. `"Google error: Invalid API key."`. valid responses are
unchanged.
adds `test/unit_test/agent/tools/test_google_unit.py` covering the
error-response path (asserts the real message is surfaced, no
`KeyError`) and the normal result path, mirroring the existing
`test_googlescholar.py` pattern (skips when the `serpapi` SDK is
absent).
---------
Co-authored-by: Yaroslav98214 <diakovichyaroslav30@gmail.com>
### Summary
`agent/tools/github.py` indexes `response["items"]` directly after
`requests.get(...).json()`. the github search api returns `{"message":
...}` **without** an `items` key on realistic conditions — a rate limit
(403/429; this tool sends no auth token, so the unauthenticated ~10
req/min search limit is easy to hit) or an invalid query (422). that
raised `KeyError('items')`, which the tool's retry loop then surfaced to
the model as the opaque `"GitHub error: 'items'"` instead of the real
reason.
this guards the missing key and raises the api's actual `message` (with
a clear fallback) into the existing retry/`_ERROR` path, so the model
sees e.g. `"GitHub error: API rate limit exceeded ..."`. valid responses
are unchanged.
adds `test/unit_test/agent/tools/test_github_unit.py` covering the
rate-limit response (asserts the real message is surfaced, no
`KeyError`) and the normal result path.
Co-authored-by: Yaroslav98214 <diakovichyaroslav30@gmail.com>
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Fixes#16855.
## Problem
`ListDatasetReq` declares `include_parsing_status: bool = False` but
`dataset_api_service.list_datasets` ignored the flag, so callers passing
`include_parsing_status=true` got `null` for every parsing-status field.
The helper that produces the counts
(`DocumentService.get_parsing_status_by_kb_ids`) already existed and
returned the documented shape; it just wasn't being called.
## Fix
Read the flag in `list_datasets`, call the helper only when truthy, and
attach per-kb counts to each record before serialisation. The flag
follows the same string-coercion idiom already used for `desc` in this
function.
When the flag is false or absent, no new field is added to the response,
so the change is non-breaking and additive. Existing callers see
byte-identical responses.
## Tests
New file
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(7 tests, all passing):
-
`test_list_datasets_without_include_parsing_status_does_not_call_helper`
- `test_list_datasets_with_include_parsing_status_true_attaches_counts`
- `test_list_datasets_with_include_parsing_status_string_true`
- `test_list_datasets_with_include_parsing_status_false_skips_helper`
-
`test_list_datasets_with_include_parsing_status_string_false_skips_helper`
-
`test_list_datasets_with_empty_kb_list_skips_helper_even_when_flag_true`
-
`test_list_datasets_with_include_parsing_status_missing_kb_gets_empty_dict`
```
$ uv run pytest test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py -v
============================== 7 passed in 0.19s ==============================
```
## Files changed
- `api/apps/services/dataset_api_service.py` (+15)
-
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(new)
## Follow-up (out of scope)
Register the `parsing_status` field in the OpenAPI response schema at
`api/apps/restful_apis/dataset_api.py` so Swagger / generated clients
document it.
Co-authored-by: Harsh23Kashyap <harsh@example.com>
### What problem does this PR solve?
The Elasticsearch and Infinity metadata push-down translators are meant
to be
interchangeable pre-filters that both mirror the in-memory `meta_filter`
fallback — the shared test section is even titled
*"is_pushdown_supported
pre-check (same logic for both backends)"*. But the two
`is_pushdown_supported`
implementations diverge:
- `common/metadata_es_filter.py` defines
`MULTIVALUE_UNSAFE_NEGATIVE_OPS = frozenset({"≠", "not in"})` and
refuses
push-down for those operators.
- `common/metadata_infinity_filter.py` has **no such guard** and pushes
them
down.
**Why the guard exists:** `meta_fields.<key>` can hold a JSON array, and
the
in-memory `meta_filter` matches a document when **any** of its values
satisfies
the predicate (per-value-bucket semantics). A document whose `tag` is
`[a, b]`
therefore still matches `tag ≠ a` — bucket `b` satisfies it. The
Infinity
push-down emits `NOT JSON_CONTAINS(meta_fields, '$.tag', '"a"')`, which
means
*"the array contains no `a` at all"*, so it **silently drops** that
document.
Same divergence for `not in`. The result: `tag ≠ a` / `tag not in (...)`
under-counts results for any document that has the excluded value
alongside
others, but only on the Infinity backend.
This is on a live production path:
`DocMetadataService._filter_doc_ids_by_metadata_infinity`
(`api/db/services/doc_metadata_service.py:948`) calls this exact
`is_pushdown_supported` as the sole gate before building the Infinity
SQL,
mirroring the ES branch at line 880 which uses the guarded ES version.
Reproduction (both real modules, no services needed):
```python
>>> from common import metadata_es_filter as es, metadata_infinity_filter as inf
>>> f = [{"op": "≠", "key": "tag", "value": "a"}]
>>> es.is_pushdown_supported(f), inf.is_pushdown_supported(f)
(False, True) # ES falls back to in-memory (correct); Infinity pushes down (wrong)
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
Add the same `MULTIVALUE_UNSAFE_NEGATIVE_OPS` set to
`metadata_infinity_filter` and reject those operators in
`is_pushdown_supported`, so a single such filter forces the whole
request to
the in-memory path — the only place the per-bucket semantics are
reproduced.
`not contains` is intentionally still allowed, matching the ES backend
(`all(not contains)` == `not any(contains)`, which the push-down
expresses
correctly on multi-valued fields). The `≠` / `not in` translators
themselves
are unchanged — they remain correct for the in-memory-fallback path;
only the
push-down eligibility gate is fixed.
### Testing
- Confirmed `is_pushdown_supported([{op}])` now returns `False` for `≠`
and
`not in` on **both** backends (previously ES=False, Infinity=True).
- Added `test_pushdown_check_rejects_multivalue_unsafe_negative_ops`,
which
asserts both backends reject these ops. Confirmed red→green: it fails
against
the pre-fix Infinity module, passes after.
- `ruff check` / `ruff format --check` clean.
### Note on overlap
Open PR #16833 also edits
`test/unit_test/common/test_metadata_filter.py`, but
only **appends** at the end of the file (line 640+) and changes
`metadata_es_filter.py` / `metadata_utils.py`, not the Infinity module —
no
overlap with this change.
### Disclosure
AI-assisted (Claude Code): the divergence was surfaced by an AI-assisted
review
pass, but I independently reproduced it against the real modules,
confirmed the
production call path, and verified the fix and test before submitting.
Signed-off-by: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Problem
Parsing a Q&A `.csv` can splice unrelated text into the wrong answers
(reported in #16791).
### Root cause
The `.csv` branch of `rag/app/qa.py`'s `chunk()` builds records with
`csv.reader(lines, delimiter=delimiter)` (default `quotechar='"'`), but
then indexes `lines[i]` by the reader's *record* index in `answer +=
"\n" + lines[i]`. When a line's field opens with a `"`, `csv.reader`
treats it as an unclosed quoted field and merges several physical lines
into one record. From there the record index permanently desyncs from
the physical line numbers, so `lines[i]` returns the wrong line and
unrelated Q&A content gets appended to the wrong answer.
### Reproduction (stdlib only)
```python
import csv
lines = 'Q1,A1\n"quoted answer start\ncontinues here,extra\nQ2,A2\n'.split("\n")
list(csv.reader(lines, delimiter=","))
# record 1 swallows 3 physical lines: ['quoted answer startcontinues here,extraQ2,A2']
# -> the reader index no longer matches lines[i]
list(csv.reader(lines, delimiter=",", quoting=csv.QUOTE_NONE))
# one physical line per record; index stays aligned
```
### Fix
Pass `quoting=csv.QUOTE_NONE` so one physical line maps to one record,
keeping the reader index aligned with `lines[i]` (the surrounding code
already relies on that 1:1 mapping).
Fixes#16791.
---------
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Table structure recognition rows, columns, headers, and spans are
produced in cropped table image coordinates, while OCR boxes are matched
later in page-cumulative coordinates. Comparing those boxes without
normalization can skip or misassign table row and column metadata.
Closes#16992.
### What is changed?
- Map TSR components from cropped or rotated table-image coordinates
back into page-cumulative coordinates before matching OCR boxes.
- Reuse one inverse rotation transform for rotated OCR boxes and TSR
components.
- Keep TSR layout ids in the same `table-N` form used by table OCR
boxes.
- Sort columns by mapped page x-coordinate after coordinate
normalization.
- Add focused unit coverage for page offsets, zoom scaling, and
90/180/270 degree rotated tables.
### Type of change
- [x] Bug fix
- [x] Test coverage
### How has this been tested?
- `uv run --group test pytest
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run --no-sync --group test pytest
--confcutdir=test/unit_test/deepdoc/parser
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run ruff check deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `uv run --no-sync python -m py_compile deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `git diff --check`
A later dependency-sync attempt was blocked while resolving the
`en-core-web-sm` wheel from GitHub, and the repository-level unit-test
conftest can try to download missing NLTK `wordnet` data when it is not
already present locally. The focused parser test above does not require
that data fixture.
---------
Co-authored-by: zq <zhouquan1511@163.com>
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
