Commit Graph

4 Commits

Author SHA1 Message Date
Jack
0403d19b5a fix: honor parser params and image VLM system_prompt in Go ingestion (#17334)
## Summary
Fix the Go ingestion pipeline so that several parser setup switches and
the
image VLM prompt are actually honored end-to-end (previously the DSL
fields
existed but the Go code never read them).

- **DOCX** (`docx_parser.go`, `docx_postprocess.go`): read `remove_toc`
and
  `remove_header_footer`; apply to both JSON and markdown output paths
  (outline-based TOC removal with a text-heuristic fallback, plus
  header/footer section filtering).
- **HTML** (`html_parser.go`, `html_postprocess.go`, `text_toc.go`):
read
`remove_header_footer` (pre-parse strip of `<header>`/`<footer>` and
ARIA
`banner`/`contentinfo`) and `remove_toc` (post-parse
`remove_contents_table`
  heuristic).
- **Markdown** (`markdown_parser.go`): read `flatten_media_to_text` and
force
  media blocks to text when enabled.
- **Image VLM** (`media_dispatch.go`): read `system_prompt` instead of
`prompt`
  so the user-configured image VLM prompt is no longer silently dropped
  (`prompt` remains the video family key).

All flags are wired through `ConfigureFromSetup`, which the dispatch
layer
already invokes for every family, so the behavior is live rather than
dead code.

## Test plan
- New unit tests: `docx_postprocess_test.go`, `html_parser_test.go`,
`text_toc_test.go`, `markdown_parser_test.go`, `media_dispatch_test.go`.
- `bash build.sh --test ./internal/parser/parser/...
./internal/ingestion/component/...`

## Notes
- The `File` component is excluded from this migration scope.
- Relates to the Python→Go parity diff (Parser 1.8–1.11, 1.15).
2026-07-24 14:42:26 +08:00
Jin Hai
d19a036cda Go: add context to lots of interface (#17253)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-22 22:30:57 +08:00
Jin Hai
b8d06d02e6 Go: add stats (#17061)
### Summary

Add LLM token stats framework

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-18 21:02:07 +08:00
qinling0210
d549194562 Implement builtin chunk method as ingestion pipeline in GO (#16822)
### Summary

Implement builtin chunk mehtod as ingestion pipeline in GO
2026-07-13 13:51:40 +08:00