## Summary
Remediates four CVEs in `lxml` and `lxml_html_clean` by upgrading
transitive dependency constraints and bumping `crawl4ai` to unblock the
resolution.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-41066 | HIGH | lxml | 5.4.0 | 6.1.0 |
| CVE-2026-49825 | HIGH | lxml_html_clean | 0.4.3 | 0.4.5 |
| CVE-2026-28348 | MEDIUM | lxml_html_clean | 0.4.3 | 0.4.5 |
| CVE-2026-28350 | MEDIUM | lxml_html_clean | 0.4.3 | 0.4.5 |
- Bumped `crawl4ai` from `>=0.9.0,<0.9.1` to `>=0.9.2,<0.9.3`, version
0.9.2 relaxes lxml constraint to `>=5.3,<7`. The strict `<0.9.3` upper
bound on crawl4ai is maintained per project convention
- Added `inscriptis>=2.7.3` to `constraint-dependencies`, version 2.7.3
relaxes lxml constraint to `>=5.4.0,<6.2.0`
- Added `lxml>=6.1.0` to `constraint-dependencies`
- Added `lxml_html_clean>=0.4.5` to `constraint-dependencies`
`exclude-dependencies` entry for `unclecode-litellm` (still declared by
crawl4ai 0.9.2) continues to prevent the namespace collision that
previously crashed RAGFlow.
## Verification
- `lxml` is used widely across the codebase for HTML/XML parsing. The
5.x → 6.x upgrade maintains API compatibility for standard usage
(`lxml.html`, `lxml.etree`).
- `lxml_html_clean` API is unchanged between 0.4.3 and 0.4.5.
- `crawl4ai` 0.9.2 is a patch bump from 0.9.0 with no API changes
affecting RAGFlow's usage.
- `inscriptis` 2.7.3 is a patch bump from 2.7.0.
## Testing
- Full unit test suite passes
- `uv sync` resolves cleanly
- Verified `crawl4ai` imports without litellm namespace collision
## Summary
- Root cause: Pipeline.Run keys the eino checkpoint by taskID only.
Resuming a failed run after the user edits the pipeline DSL recompiles a
graph with different topology, but the old checkpoint (bound to the
previous graph's node ids / wiring) is restored, causing eino to error.
- Fix: fingerprint the DSL file (full canvas DSL) and the runtime
override_params, persisted next to the eino checkpoint. On resume, if
either fingerprint differs, discard the stale checkpoint + interrupt
marker and re-run from scratch. The warning log distinguishes a DSL-file
edit from a runtime-override edit.
## Test plan
- TestPipelineRunResumableDSLChanged: editing the DSL between runs
discards the checkpoint and re-runs from scratch.
- TestPipelineRunResumableOverrideChanged: editing only the runtime
override does the same.
- TestClassifyDSLChange: unit-tests the mismatch-reason classifier.
- bash build.sh --test ./internal/ingestion/pipeline/... passes.
## Notes
- Component-code changes (e.g. a component's output contract) are not
covered by the DSL fingerprint; that is a separate, smaller-blast-radius
gap noted in code comments.
---------
Signed-off-by: xugangqiang <xugangqiang@hotmail.com>
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Description
This PR fixes Title Case capitalization matching for Table of Contents
items in `README.md`.
## Details
Updated TOC entries (`Build a Docker image` -> `Build a Docker Image`,
`Launch service from source` -> `Launch Service from Source`) to match
section headers.
Co-authored-by: ferkans-amir <amir.rezaei@tu-berlin.de>
## Summary
Fix the Go ingestion pipeline so that several parser setup switches and
the
image VLM prompt are actually honored end-to-end (previously the DSL
fields
existed but the Go code never read them).
- **DOCX** (`docx_parser.go`, `docx_postprocess.go`): read `remove_toc`
and
`remove_header_footer`; apply to both JSON and markdown output paths
(outline-based TOC removal with a text-heuristic fallback, plus
header/footer section filtering).
- **HTML** (`html_parser.go`, `html_postprocess.go`, `text_toc.go`):
read
`remove_header_footer` (pre-parse strip of `<header>`/`<footer>` and
ARIA
`banner`/`contentinfo`) and `remove_toc` (post-parse
`remove_contents_table`
heuristic).
- **Markdown** (`markdown_parser.go`): read `flatten_media_to_text` and
force
media blocks to text when enabled.
- **Image VLM** (`media_dispatch.go`): read `system_prompt` instead of
`prompt`
so the user-configured image VLM prompt is no longer silently dropped
(`prompt` remains the video family key).
All flags are wired through `ConfigureFromSetup`, which the dispatch
layer
already invokes for every family, so the behavior is live rather than
dead code.
## Test plan
- New unit tests: `docx_postprocess_test.go`, `html_parser_test.go`,
`text_toc_test.go`, `markdown_parser_test.go`, `media_dispatch_test.go`.
- `bash build.sh --test ./internal/parser/parser/...
./internal/ingestion/component/...`
## Notes
- The `File` component is excluded from this migration scope.
- Relates to the Python→Go parity diff (Parser 1.8–1.11, 1.15).
## Summary
Adds construction-time parameter validation to the ingestion
`ParserComponent` (mirroring the applicable subset of Python
`ParserParam.check()`), fixes a family-mapping mismatch that silently
skipped `output_format` validation and setup configuration for
image/audio files, and aligns the no-CGO parser stubs with the CGO
variants by threading `context.Context` through `ParseWithResult`.
### What problem does this PR solve?
Fix Infinity compatibility issues in knowledge compilation.
This change:
- Stores compilation source ID lists as JSON arrays in Infinity.
- Parses JSON array fields when reading compiled documents.
- Uses `json_contains` for filtering JSON array fields.
- Adds the missing `name` column to the Infinity mapping.
- Updates dataset navigation KNN search to use the unified
`MatchDenseExpr` interface.
- Handles unavailable embeddings without querying an invalid `q_0_vec`
field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Summary
**Issue**
When calling Infinity under concurrent load, one shared connection was
reused across concurrent operations, it just hangs.
**Solution**
Avoid hangs with pooled connections and bounded timeouts
## Summary
Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.
## What changed
### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.
### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.
### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).
### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.
## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
