Jack
9b0719fa94
fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) ( #17419 )
...
## Summary
Continuation of the Python→Go ingestion pipeline migration (File →
Parser → Chunker → Extractor → Tokenizer). Fixes cover Parser, Chunker,
and Tokenizer gaps identified. Fix page number (0-indexed and 1-index
mixed before fix; use 1-indexed after fix) and chunk order issues.
### Parser
- **Slides TCADP (1.7):** `pptx_tcadp.go` + TCADP branch in
`pptx_parser.go`/`ppt_parser.go` — PowerPoint files now support
`parse_method="tcadp"` via the TCADP cloud service, matching the
spreadsheet-family TCADP pattern. PPT containers pass `"PPT"` as
fileType (not hardcoded `"PPTX"`).
- **Audio default output_format (2.11):** `defaultSetups()` audio
default changed from `"text"` to `"json"`, aligning with Python
`parser.py:232` and `AllowedOutputFormat["audio"]={"json"}`.
- **PDF VLM enhancement (1.1):** `maybeDispatchPDFVisionEnhancement` in
`pdf_vision_dispatch.go` enriches image/table items with IMAGE2TEXT
model descriptions after PDF parsing, mirroring Python
`enhance_media_sections_with_vision`. Semaphore fix: acquire before
goroutine start to prevent unbounded goroutine creation.
- **json family (2.3):** reclassified as Keep Go — `json_parser.go` is a
functional enhancement, not a parity gap.
- **page number:** changed from "mixed use of 1-indexed & 0-indexed" to
"1-indexed"
### Chunker
- **BULLET_PATTERN fallback (1.7):** 4th-level fallback in
`resolveTitleLevels` (`title.go`) detects bullet/numbered-list patterns
(Chinese legal, numbering, English) when outline + regex levels produce
only bodyLevel. Guarded by `allBodyLevel` to never override existing
structure.
- **Tag/One chunker fields (1.8):** `tag.go` sets `TopInt` from source
row index; `one.go` preserves `Positions`/`PDFPositions` from source
items. TSV multi-line RowNum fix: tracks `contentStart` for correct row
attribution.
- **Overlapped_percent normalization (2.6):**
`NormalizeOverlappedPercent` in `schema/chunker.go` mirrors Python
`common/float_utils.py:50-58` — accepts `[0,1)` fraction or `[0,90]`
percent, normalizes to canonical `[0,90]`.
- **Paragraph splitting (2.7):** aligned to Python flow `naive_merge` —
`CRLF` normalization, `splitKeepingDelimiter` preserves sentence
delimiters, single-section merge with token-budget-governed chunking.
- **chunk order:** sort by reading order
### Tokenizer
- **Phantom chunk filtering (Omission 2):** `isPhantomChunk` + filter
loop in `chunksFromTokenizerUpstream` skips zero-value ChunkDocs.
- **Batch size env var (Omission 3):** `embeddingBatchSize()` reads
`TOKENIZER_EMBEDDING_BATCH_SIZE`, defaults to 16.
- **Summary empty check (Diff 5):** `TrimSpace(s) != ""` → `s != ""`,
matching Python truthy check.
- **chunk_order_int all paths (Diff 8):** set unconditionally before
full_text/embedding branching.
- **Timeout default (Diff 10):** `600s` → `60s`, matching Python
`@timeout(60)`.
- **Small maxTokens truncation (Diff 14):** `truncateForEmbedding`
returns `""` when `maxTokens <= 10`, matching Python.
### Code review fixes
- Semaphore acquire moved before goroutine in `pdf_vision_dispatch.go`
(concurrency control)
- Context propagation in `pptx_tcadp.go` (cancellation support)
- Test resolver leak fix in `media_dispatch_test.go` (defer restore)
- Migration history comments removed per AGENTS.md
## Test plan
```
bash build.sh --test ./internal/parser/parser/... ./internal/ingestion/component/...
```
## Notes
- Migration diff tracking: `docs/migration_python_go_diff.md`
- Remaining gaps: Extractor component only (21 items)
2026-07-28 11:12:52 +08:00
Jin Hai
53e83dcadf
Go: add context, part4 ( #17381 )
...
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-07-26 18:31:56 +08:00
Jin Hai
bdfc3ada41
Go: add context ( #17314 )
...
### Summary
As title.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-07-24 16:47:12 +08:00
Jack
965590ccbe
Refactor: dataset/document/file service ( #17071 )
...
### Summary
Refactor dataset.go document.do file.go file2document.go in
internal/service.
2026-07-20 09:48:24 +08:00
Jack
10c00a9614
Feat: add built in DSL file API ( #17003 )
...
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
2026-07-18 11:00:08 +08:00
Jack
047c714ffb
Feat: ingestion cleanup ( #16953 )
...
### Summary
1. Remove dead code (replaced by builtin ingestion pipeline)
2. Refactor (move document parsing progress from http api into ingestion
executor)
2026-07-16 11:05:26 +08:00
Jack
e543ff02c4
Feat: nats message processing refactor and support built in DSL ( #16873 )
2026-07-15 14:53:16 +08:00
Jack
cfacaccad7
Refactor: message processing ( #16852 )
...
### Summary
1. refactor message processing
2. delete un-used componentIndexMap
3. unfold (delete) internal/ingestion/task/task_handler.go
2026-07-13 16:32:34 +08:00
Jack
3c0f8fc192
Refactor: refactor dataflow_service.go and guard nats message re-delivery ( #16826 )
...
### Summary
1. refactor dataflow_service.go
2. guard nats message re-delivery
3. support document parse cancelling & re-run
2026-07-13 11:08:04 +08:00
Jack
5e60fcec9f
Refactor: Refine ingestion task state transitions ( #16814 )
...
### Summary
Refine ingestion task state transitions
2026-07-10 22:47:51 +08:00
Jack
7db39822db
Feature: user select pipeline support ( #16788 )
...
### Summary
Feature: user select pipeline support
2026-07-10 14:30:28 +08:00
Jack
8bd62573eb
Fix: delete tasklet ( #16778 )
...
### Summary
1. Fix: delete tasklet
2. Remove panic in text processing
2026-07-09 17:31:28 +08:00
Jack
8ac80284c4
Feat: add embedding support ( #16769 )
...
### Summary
Feat: add embedding support
2026-07-09 16:24:39 +08:00
Jack
0dd0ac06f8
Feature: task executor migration to go ( #16549 )
...
### Summary
Feature: Integrate parser
2026-07-08 19:08:11 +08:00
