fix(parser/email): align metadata/text_html emission with Python flow contract (#18005)

Aligns the Go EML parser (`internal/parser/parser/email_parser.go`) with the Python flow parser (`rag/flow/parser/parser.py:_email`) on two structural points so the Go path is a faithful drop-in for the Python path on `.eml` inputs.
This commit is contained in:
Jack
2026-08-07 21:57:51 +08:00
committed by GitHub
parent 4b4a6e72f0
commit 4f15e261cd
4 changed files with 439 additions and 9 deletions

View File

@@ -233,6 +233,13 @@ class RAGFlowHtmlParser:
def _token_count(cls, text):
if not text:
return 0
# FIXME: The identity of the tokenizer behind `rag_tokenizer.tokenize`
# (used here to size HTML chunks) is NOT confirmed. The earlier assumption
# that it resolves to `infinity.rag_tokenizer.RagTokenizer` is incorrect.
# Verify what `rag_tokenizer.tokenize` actually resolves to before claiming
# Parser and TokenChunker use different tokenizers. For reference,
# TokenChunker uses `num_tokens_from_string` (common/token_utils.py) which
# is tiktoken cl100k_base. See PARSER_ALIGNMENT_HANDOFF.md §3.2.
tks_str = rag_tokenizer.tokenize(text)
return len(tks_str.split(" ")) if tks_str else 0