mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-11 01:41:26 +08:00
fix(parser/email): align metadata/text_html emission with Python flow contract (#18005)
Aligns the Go EML parser (`internal/parser/parser/email_parser.go`) with the Python flow parser (`rag/flow/parser/parser.py:_email`) on two structural points so the Go path is a faithful drop-in for the Python path on `.eml` inputs.
This commit is contained in:
@@ -233,6 +233,13 @@ class RAGFlowHtmlParser:
|
||||
def _token_count(cls, text):
|
||||
if not text:
|
||||
return 0
|
||||
# FIXME: The identity of the tokenizer behind `rag_tokenizer.tokenize`
|
||||
# (used here to size HTML chunks) is NOT confirmed. The earlier assumption
|
||||
# that it resolves to `infinity.rag_tokenizer.RagTokenizer` is incorrect.
|
||||
# Verify what `rag_tokenizer.tokenize` actually resolves to before claiming
|
||||
# Parser and TokenChunker use different tokenizers. For reference,
|
||||
# TokenChunker uses `num_tokens_from_string` (common/token_utils.py) which
|
||||
# is tiktoken cl100k_base. See PARSER_ALIGNMENT_HANDOFF.md §3.2.
|
||||
tks_str = rag_tokenizer.tokenize(text)
|
||||
return len(tks_str.split(" ")) if tks_str else 0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user