Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
## Summary
- The `eng` flag passed to `removeTOCWord`/`removeContentsTable` was
hardcoded `false`, so English documents used the CJK 3-char TOC prefix
instead of Python's 2-word English prefix. This caused English tables of
contents to be under-deleted (left in indexed text) or over-deleted
(body paragraphs sharing the 3-char prefix dropped).
- `eng` is now derived from the parsed item content via `isEnglishItems`
(a port of Python `is_english`/`_is_english`), mirroring Python's
content heuristic.
## Changes
- `internal/parser/parser/text_toc.go`: add
`isEnglishItems`/`isEnglishTexts` (ASCII-ratio >80% ⇒ English,
fullmatch-anchored regex).
- `internal/parser/parser/docx_parser.go`: pass
`isEnglishItems(sections)` / `isEnglishItems(lineItems)` to
`removeTOCWord` (json + markdown paths).
- `internal/parser/parser/html_parser.go`: pass `isEnglishItems(items)`
to `removeContentsTable`.
- `docs/migration_python_go_diff.md`: close the 1.9/1.10 `eng` residual
and resolve the contradictory Parser 2.11 "Partially fixed" note.
## Notes
- `remove_toc` is a Parser-stage, DSL-configured feature (`remove_toc:
true/false` in the parser family setup). This change only fixes the
internal English/CJK prefix decision; it does not alter the pipeline or
the DSL contract. `remove_header_footer` (precise match) is unaffected.
- `eng` is auto-derived from content rather than exposed as a new config
knob, to stay faithful to Python behavior.
## Test plan
- `CGO_ENABLED=0 go test ./internal/parser/parser/ -run
'TestRemoveContentsTable|TestIsEnglishTexts|TestRemoveTOCWordEnglishDetection'`
- Added `TestIsEnglishTexts` (ASCII-ratio cases) and
`TestRemoveTOCWordEnglishDetection` (English TOC no longer over-deletes
"Chapter 2 Method").
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
Fix the Go ingestion pipeline so that several parser setup switches and
the
image VLM prompt are actually honored end-to-end (previously the DSL
fields
existed but the Go code never read them).
- **DOCX** (`docx_parser.go`, `docx_postprocess.go`): read `remove_toc`
and
`remove_header_footer`; apply to both JSON and markdown output paths
(outline-based TOC removal with a text-heuristic fallback, plus
header/footer section filtering).
- **HTML** (`html_parser.go`, `html_postprocess.go`, `text_toc.go`):
read
`remove_header_footer` (pre-parse strip of `<header>`/`<footer>` and
ARIA
`banner`/`contentinfo`) and `remove_toc` (post-parse
`remove_contents_table`
heuristic).
- **Markdown** (`markdown_parser.go`): read `flatten_media_to_text` and
force
media blocks to text when enabled.
- **Image VLM** (`media_dispatch.go`): read `system_prompt` instead of
`prompt`
so the user-configured image VLM prompt is no longer silently dropped
(`prompt` remains the video family key).
All flags are wired through `ConfigureFromSetup`, which the dispatch
layer
already invokes for every family, so the behavior is live rather than
dead code.
## Test plan
- New unit tests: `docx_postprocess_test.go`, `html_parser_test.go`,
`text_toc_test.go`, `markdown_parser_test.go`, `media_dispatch_test.go`.
- `bash build.sh --test ./internal/parser/parser/...
./internal/ingestion/component/...`
## Notes
- The `File` component is excluded from this migration scope.
- Relates to the Python→Go parity diff (Parser 1.8–1.11, 1.15).
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
