mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-31 04:59:24 +08:00
989529c00a2f1cf4f3f797aefbc7b87ec437e19d
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Jack
|
76aaecc284 |
fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
|
||
|
Jack
|
9b0719fa94 |
fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)
## Summary
Continuation of the Python→Go ingestion pipeline migration (File →
Parser → Chunker → Extractor → Tokenizer). Fixes cover Parser, Chunker,
and Tokenizer gaps identified. Fix page number (0-indexed and 1-index
mixed before fix; use 1-indexed after fix) and chunk order issues.
### Parser
- **Slides TCADP (1.7):** `pptx_tcadp.go` + TCADP branch in
`pptx_parser.go`/`ppt_parser.go` — PowerPoint files now support
`parse_method="tcadp"` via the TCADP cloud service, matching the
spreadsheet-family TCADP pattern. PPT containers pass `"PPT"` as
fileType (not hardcoded `"PPTX"`).
- **Audio default output_format (2.11):** `defaultSetups()` audio
default changed from `"text"` to `"json"`, aligning with Python
`parser.py:232` and `AllowedOutputFormat["audio"]={"json"}`.
- **PDF VLM enhancement (1.1):** `maybeDispatchPDFVisionEnhancement` in
`pdf_vision_dispatch.go` enriches image/table items with IMAGE2TEXT
model descriptions after PDF parsing, mirroring Python
`enhance_media_sections_with_vision`. Semaphore fix: acquire before
goroutine start to prevent unbounded goroutine creation.
- **json family (2.3):** reclassified as Keep Go — `json_parser.go` is a
functional enhancement, not a parity gap.
- **page number:** changed from "mixed use of 1-indexed & 0-indexed" to
"1-indexed"
### Chunker
- **BULLET_PATTERN fallback (1.7):** 4th-level fallback in
`resolveTitleLevels` (`title.go`) detects bullet/numbered-list patterns
(Chinese legal, numbering, English) when outline + regex levels produce
only bodyLevel. Guarded by `allBodyLevel` to never override existing
structure.
- **Tag/One chunker fields (1.8):** `tag.go` sets `TopInt` from source
row index; `one.go` preserves `Positions`/`PDFPositions` from source
items. TSV multi-line RowNum fix: tracks `contentStart` for correct row
attribution.
- **Overlapped_percent normalization (2.6):**
`NormalizeOverlappedPercent` in `schema/chunker.go` mirrors Python
`common/float_utils.py:50-58` — accepts `[0,1)` fraction or `[0,90]`
percent, normalizes to canonical `[0,90]`.
- **Paragraph splitting (2.7):** aligned to Python flow `naive_merge` —
`CRLF` normalization, `splitKeepingDelimiter` preserves sentence
delimiters, single-section merge with token-budget-governed chunking.
- **chunk order:** sort by reading order
### Tokenizer
- **Phantom chunk filtering (Omission 2):** `isPhantomChunk` + filter
loop in `chunksFromTokenizerUpstream` skips zero-value ChunkDocs.
- **Batch size env var (Omission 3):** `embeddingBatchSize()` reads
`TOKENIZER_EMBEDDING_BATCH_SIZE`, defaults to 16.
- **Summary empty check (Diff 5):** `TrimSpace(s) != ""` → `s != ""`,
matching Python truthy check.
- **chunk_order_int all paths (Diff 8):** set unconditionally before
full_text/embedding branching.
- **Timeout default (Diff 10):** `600s` → `60s`, matching Python
`@timeout(60)`.
- **Small maxTokens truncation (Diff 14):** `truncateForEmbedding`
returns `""` when `maxTokens <= 10`, matching Python.
### Code review fixes
- Semaphore acquire moved before goroutine in `pdf_vision_dispatch.go`
(concurrency control)
- Context propagation in `pptx_tcadp.go` (cancellation support)
- Test resolver leak fix in `media_dispatch_test.go` (defer restore)
- Migration history comments removed per AGENTS.md
## Test plan
```
bash build.sh --test ./internal/parser/parser/... ./internal/ingestion/component/...
```
## Notes
- Migration diff tracking: `docs/migration_python_go_diff.md`
- Remaining gaps: Extractor component only (21 items)
|
||
|
Jack
|
76acd499d4 |
Fix: use independent fixtures in ClampsOverlappedPct test (#17416)
## Problem `TestMergeByTokenSizeFromJSON_ClampsOverlappedPct` reused a single `items` fixture across four calls to `mergeByTokenSizeFromJSON`. `mergeByTokenSizeFromJSON` mutates its `perItem` argument in place and returns the same backing array (token.go: `perItem[idx] = merged`). As a result: - The 2nd and later calls merged already-merged chunks instead of the original input. - Because the returned slice aliases the input, later calls silently overwrote the earlier results (`at100`, `at0`, ...). - `reflect.DeepEqual(at100, at150)` was therefore vacuously true — the test was a **false positive** that never actually exercised the clamp. It would still pass even if the clamp were broken. This is exactly the defect flagged in the review comment on PR #17396. ## Fix Add a `clampOverlapFixture()` factory and build a fresh fixture for every call, so that 150 / 1e300 / -5 / -1e300 are applied to the original input. ## Verification - The test passes after the fix. - When the clamp logic was temporarily disabled, the test **failed** (panic at token.go:768 — the negative index produced by an out-of-range pct), proving the fixed test is no longer a false positive and can catch a clamp regression. ## Scope Test file only. No production code change (verified `token.go` is identical to `upstream/main`). Refs: review comment on PR #17396. |
||
|
Jack
|
a6a67c5ece |
Chunker: port Python overlapped_percent normalization and BULLET_PATTERN title fallback (#17396)
## Summary Closes two Chunker migration gaps documented in `docs/migration_python_go_diff.md` (diffs **2.6** and **1.7**), improving parity with the Python ingestion pipeline. This is the code portion of commit `261e1fd0b` on branch `fix/batch-3-4`; the migration document itself is tracked separately (untracked in this commit). ### Diff 2.6 — `overlapped_percent` missing Python normalization Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`) accepts a `[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it by 100, `int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a raw `[0,90]` percentage and **rejected** out-of-range input, so a Python config passing `0.1` (meaning 10%) silently produced ~0% overlap. Added `normalizeOverlappedPercent` (`internal/ingestion/component/chunker/common.go`) mirroring the Python helper: - parses numbers and numeric strings (mirrors Python `float()`; bad / `NaN` / `Inf` → `0`), - `0 < v < 1 → v *= 100`, - `int()` truncation, - clamps to `[0,90]`. Wired into `tokenChunkerParam.Update` (`token.go`); the merge math (`token.go:705`, `(100-x)/100`) already matched Python. `TokenChunkerParam.Validate` (`schema/chunker.go`) now only guards direct struct construction. Tests: `TestNormalizeOverlappedPercent`, extended `TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp inputs), new `TestTokenChunker_NormalizesOverlappedPercent`. The existing reject-test cases for `<0` / `>90` were removed because they are now normalized/clamped (Python parity). ### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback `resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when outline + regex + layout all yield body level, `bulletsCategory` selects the best-matching bullet-pattern group (Chinese legal / numbering / Chinese numbering / English legal — mirroring `rag/nlp/__init__.py:258-320`) and assigns structural levels. Guarded by `allBodyLevel` so it never overrides an existing outline/regex level. Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests). ### Incidental test adjustments included in the commit - `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to reflect the post-normalization 30% semantics. - `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` → `LoadFromIngestionTask(ctx, task)` for the new context-first signature. ## Verification `bash build.sh --test ./internal/ingestion/component/...` — chunker + schema suites pass, no regression (CGO build). ## Migration doc reference `docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked **Fixed** for these changes. |
||
|
Jin Hai
|
f53518c110 |
Go: add context, part5 (#17392)
Signed-off-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Jack
|
554925b583 |
Fix(go): align ingestion pipeline with Python (parser/media dispatch + PDF coordinate chain + Chunker) (#17349)
## Summary Aligns the Go ingestion pipeline with the Python implementation, closing several behavioral gaps found during the Python→Go migration (tracked in `docs/migration_python_go_diff.md`). Covers parser/media dispatch alignment, the PDF coordinate-chain (preview images, outline→title, chunk coordinate finalization), and the Chunker Token/QA batches below. Commits are grouped as follows. ### 1. Fix parser params ( |
||
|
Jin Hai
|
bdfc3ada41 |
Go: add context (#17314)
### Summary As title. --------- Signed-off-by: Jin Hai <haijin.chn@gmail.com> |
||
|
euvre
|
a5aa9489f2 | fix(chunker): glue delimiter to preceding segment instead of emitting it as a standalone chunk (#17168) | ||
|
Jack
|
8669c469d5 |
fix(ingestion): align laws DSL with Python — heading fallback, colon-title, short-line filter, remove_toc, and image extension mapping (#17200)
## Summary
This PR aligns the Go ingestion pipeline's **Laws** DSL template with
the Python implementation by fixing heading-detection gaps, adds
image-extension support, refactors the **Extractor** component's LLM
resolution, hardens heading detection for CJK text, and makes the
Extractor accept the Python DSL prompt key names
(`sys_prompt`/`prompts`) alongside the Go names.
## Changes
### 1. Picture file-type detection (`internal/utility/file.go`)
Adds explicit mapping for common image extensions (png, jpg, jpeg, gif,
bmp, tiff, tif, webp, svg, ico, avif, heic, apng) → `FileTypeVISUAL`,
with regression tests.
### 2. Laws DSL heading-detection alignment
(`internal/ingestion/component/chunker/`)
Four fixes to `resolveTitleLevels`:
| Fix | What changed | Why |
|-----|-------------|-----|
| **DOCX `ck_type` fallback** | `ckType` field on `lineRecord`,
propagated from `ChunkDoc.CKType` in `recordsFromStructured`. When
`ck_type=="heading"`, assign `fallbackLevel`. | office_oxide extracts
DOCX heading metadata, but the info was lost before reaching the heading
detector. Word headings whose text doesn't match any regex (e.g.
"Introduction") were treated as body. |
| **`make_colon_as_title` promotion** | `isColonTitle()`: promotes lines
ending with `:`/`：` that have sentence-ending punctuation before the
colon and ≥32 runes between them. | Mirrors Python's
`make_colon_as_title` in `rag/nlp/__init__.py`. Triple guard prevents
false positives. |
| **Short/numeric line filter** | Lines with ≤1 rune or purely numeric
are pinned to body level. | Mirrors Python `tree_merge`'s filter of
`sections` where `len(...) <= 1` or `re.match(r"[0-9]+$", ...)`. |
| **PDF `remove_toc`** | `"remove_toc": true` added to the PDF parser
setup in `ingestion_pipeline_laws.json`. | The Go PDF parser already
supports TOC removal; the Book template already enables it. |
### 3. Extractor llm_id resolution
(`internal/ingestion/component/extractor.go`)
Refactored to handle both **bare tenant_model UUIDs** and **composite
model@provider** strings via the shared `resolveModelConfig`
(`dispatch_model.go`):
- **`resolveExtractorChatConfig`** — UUID path calls
`resolveModelConfigByID` directly (one DB hit); composite path goes
through `resolveModelConfig`. Added `isBareTenantModelID` pre-check for
clear errors when a UUID doesn't exist.
- **`resolveExtractorChatTarget`** — propagates resolution errors
instead of silently returning empty driver.
- **`Chat()`** — removed `driver = "dummy"` fallback. Missing driver is
now an explicit error.
- **Removed dead code**: `splitExtractorLLID`,
`findExtractorSoleActiveInstance`.
### 4. `InjectExtractorLLMID` — fallback when no user config
(`internal/common/parser_config.go`)
Injects the tenant's global default LLM into extractor components **only
when their `llm_id` is empty**. Preserves user-selected UUID or
model@provider values.
Priority: user-configured llm_id > tenant global default > error (no
silent dummy fallback).
### 5. `ResponseHeaderTimeout` increase
(`internal/entity/models/base_model.go`)
`ResponseHeaderTimeout` 60s → 120s in `NewDriverHTTPClient`. Reasoning
models with large extraction prompts can take longer than 60s to produce
the first response token.
### 6. CJK rune-aware heading detection
(`internal/ingestion/component/chunker/title.go`)
Two byte-vs-rune bugs that only manifest on CJK text:
| Fix | What changed | Why |
|-----|-------------|-----|
| **`isColonTitle` byte offset** | `body[lastPunct+1:]` →
`body[lastPunct+runeLen:]` via `utf8.DecodeRuneInString` |
`strings.LastIndexAny` returns a byte index; `+1` skips only 1 byte,
corrupting multi-byte CJK punctuation (e.g. `。` = 3 bytes) and inflating
the rune count past the 32-rune threshold → false-positive heading
promotion. |
| **Short-line filter byte count** | `len(text) <= 1` →
`utf8.RuneCountInString(text) <= 1` | Go `len` is UTF-8 bytes; a single
CJK char (3 bytes) passed the filter, but Python's `len` returns 1 →
mismatch. |
### 7. `extractor_tag.go` — log error when llm fails
When `resolveExtractorChatTarget` returned an error, `runAutoTags` will
log error.
### 8. Python DSL prompt-key compatibility
(`internal/ingestion/component/extractor.go`)
The Resume DSL template uses Python-side key names (`sys_prompt`,
`prompts`). `NewExtractorComponent` now accepts them as fallbacks
alongside the Go names:
- `system_prompt` (Go) ← `sys_prompt` (Python) as fallback
- `prompt` (Go string) ← `prompts` (Python array `[{"role","content"}]`,
takes `[0].content`) as fallback
Mirrors the alias pattern already in `internal/agent/component/llm.go`.
`resolveInputs` accepts per-call `sys_prompt` override too.
## Remaining gaps vs Python
| Gap | Scope | Impact |
|-----|-------|--------|
| **TOC removal for TXT/MD/HTML** | Python's `remove_contents_table`
works on all text formats; Go's `remove_toc` is PDF-only. | Low —
plain-text documents rarely contain structured TOCs. |
| **Regex pattern details** | Minor differences in quantifiers, missing
H5/H6 markdown patterns, missing 4-level numbering pattern. | Low — Go's
variants are stricter; DOCX headings are covered by `ck_type` fallback.
|
## Testing
- `TestHierarchyTitleChunker_CKTypeHeadingFallback` — DOCX `ck_type`
heading promotion
- `TestHierarchyTitleChunker_ColonTitlePromotion` /
`_ColonTitleShortLine_Negative` — colon-title promotion + guard
- `TestIsColonTitle_CJKEdgeCase` / `TestIsColonTitle_ASCII_NoRegression`
— CJK byte-offset fix + ASCII regression
- `TestHierarchyTitleChunker_ColonTitlePromotion_CJK_EdgeCase` — CJK
colon edge case through full pipeline
- `TestHierarchyTitleChunker_ShortSingleCJKLineFilter` — single CJK char
filtered to body
- `TestHierarchyTitleChunker_ShortNumericLineFilter` — purely numeric
lines filtered
- `TestGetFileType_ImageExtensions` / `_ExistingFormats_NoRegression` —
image extension mapping
- `TestInjectExtractorLLMID_SkipWhenUUID` / `_SkipWhenComposite` /
`_InjectWhenEmpty` — llm_id injection guard
- `TestIsBareTenantModelID` — UUID detection
- `TestResolveExtractorChatTarget_AtSplitFallback` / `_NoDriver` — @
split fallback without DB
- `TestNewExtractorComponent_SysPromptAlias` / `_PromptsArray` /
`_PromptsArray_PromptWins` / `_SystemPromptWinsOverSysPrompt` — Python
key compatibility
- `TestBuildDOCXJSONSections_List` / `_TextBox` / `_MixedWithList` —
DOCX list/text_box parsing
- Full ingestion test suite passes (chunker, pipeline, task, service,
component packages)
|
||
|
qinling0210
|
0189ca3700 |
Port PR14140 and PR16881 to GO (#17102)
### Summary Port https://github.com/infiniflow/ragflow/pull/14140/ https://github.com/infiniflow/ragflow/pull/16881 |
||
|
Jack
|
bb6b43b5c9 |
fix: docx/email parsing, extractor LLM driver, and chunker alignment (#17144)
## Summary Three groups of changes across the Go ingestion pipeline: ### 1. DOCX parsing improvements - **docx_parser.go**: Enhanced DOCX parsing with better structure extraction and media handling - **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests - **office_parsers_no_cgo.go**: Stub sync for non-CGO builds ### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding - **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding decoding for base64 and quoted-printable. Go's `mime/multipart.Reader` does not decode Content-Transfer-Encoding automatically, so attachments with `Content-Transfer-Encoding: base64` remained base64-encoded in the output. The new `decodeCTE` helper is called after reading each multipart part's raw bytes in `readMailBody`, mirroring Python's `part.get_payload(decode=True)`. - **email_parser_test.go**: Two new tests — simple base64 attachment and nested multipart/alternative with base64 attachment. ### 3. Extractor LLM driver fix + ModelDriver consolidation - **extractor.go**: Fixed a bug where the Extractor component used `ModelFactory.CreateModelDriver()`, which creates bare model instances without API keys or provider configuration. Switched to `models.GetPreconfiguredDriver()` which resolves the actual pre-configured driver from `ProviderManager`, matching the codepath used by `llm.go`. This fixes auto keyword/question extraction in DSL pipelines that require LLM calls. - **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()` from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go` use the same codepath. - **get_driver_test.go** (new): Tests for the shared driver resolution. - **llm.go**: Replaced inline driver resolution with `models.GetPreconfiguredDriver()`. ### 4. Chunker fixes and observability - **group.go** (`extractLineRecords`): Fixed to also read `markdown` and `html` payload keys — previously it only read `text`/`content`, causing GroupTitleChunker to silently return empty results for markdown-format parser output. - **common.go** (`compileDelimPattern`): Aligned with Python's `_compile_delimiter_pattern` — only backtick-wrapped delimiters produce an active regex pattern; plain delimiters are not compiled into the split regex. - **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to `"text"` on created ChunkDocs so the token-size merge path correctly identifies and merges text segments. - **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**, **hierarchy.go**: Added debug-level logging for pipeline diagnostics. - **parser_dispatch_test.go**, **group_test.go**: New tests. ## Verification - All Go tests pass: `bash build.sh --test ./internal/parser/parser/...` and `bash build.sh --test ./internal/ingestion/component/...` - Build succeeds: `bash build.sh --go` |
||
|
Jack
|
2f7c2eb53c |
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111)
## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ``` |
||
|
qinling0210
|
995e405e8c |
Support pipeline DSL modification through dataset configuration (backend) (#16991)
…end) ### Summary Support pipeline DSL modification through dataset configuration (backend) Key modification: knowledgebase.parser_config --------- Co-authored-by: yzc <yuzhichang@gmail.com> |
||
|
Zhichang Yu
|
d279aee1ff |
Go ports of workflow and chunker fixes (#16878)
Ports two Python fixes to Go: the variable_ref_patt underscore/colon fix (#16792) and the TokenChunker upstream-chunks fix (#16825). Keeps Go behavior aligned with upstream Python. |
||
|
qinling0210
|
d549194562 |
Implement builtin chunk method as ingestion pipeline in GO (#16822)
### Summary Implement builtin chunk mehtod as ingestion pipeline in GO |
||
|
Zhichang Yu
|
fb42e5531d |
Refactor: drop dead canvas runtime selector and tokenizer embedding wiring (#16809)
Two refactors on the Go port (agent-go-port): - Remove the dead per-tenant canvas runtime selector (write-only Redis scaffolding with no runtime callers) and its dependent metrics/admin code. - Move the tokenizer embedding-model id from the shared ingestion globals into a Tokenizer-scoped setup, and wire the production embedder resolver in the ingestion task package. 32 files changed, 861 insertions, 1228 deletions. |
||
|
Zhichang Yu
|
12787996d1 |
feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795)
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
|
||
|
Zhichang Yu
|
014c3f634f |
Align Go ingestion boundaries with Python (#16647)
Moves doc_id blob resolution into Parser, tightens chunker/tokenizer to Python output_format semantics, updates extractor list handling, and fixes real-template integration tests. |