renderTableHTML only fails on unsupported node kinds, which a parsed
<table> never triggers, so the degenerate flatten-children branch in
walkHTMLLeaf is unreachable. Remove it so the nested-table path matches
the top-level walkHTMLBlocks case (both skip silently when the markup is
empty).
Also tighten the inaccurate 'mirrors Python's HtmlParser' inline comment
in the same block (the two-item split is the markdown_parser.go model;
Python keeps the table once, outside the inline flow).
Stop flattening <table> into a single text blob. A <table> now emits:
1. an inlined doc_type_kwd:"text" item keeping the <table>...</table> markup
(row/column structure survives for embedding/retrieval/LLM rendering), and
2. a structured doc_type_kwd:"table" / ck_type:"table" item appended after
the walk, consumed by the downstream chunker.
Covers both top-level tables (walkHTMLBlocks) and tables nested in
div/section/article (via walkHTMLLeaf). Mirrors markdown_parser.go table
handling and Python's RAGFlowHtmlParser.
Tests:
- TestHTMLParser_TableProducesStructuredItems
- TestHTMLParser_NestedTableProducesStructuredItems
- TestHTMLParser_NestedListItemsPreserved
- TestHTMLParser_PrePreservesVerbatim
- TestHTMLParser_AlignmentGolden (compares vs html.python.golden.json, ignores
"table" via meta.accepted_divergences)
Depends on #18014 and the foundation helpers PR (align_test.go).
Files:
internal/parser/parser/html_parser.go
internal/parser/parser/html_parser_align_test.go (new)
internal/parser/parser/testdata/html.python.golden.json (new)
internal/parser/parser/testdata/html.sample.html (new)
DefaultTextCodeDelimiter and DefaultMarkdownDelimiter are shared by both
the production parsers and the alignment tests. They previously lived in
align_test.go (a _test.go file), so production text_parser.go had to keep
a private duplicate that could drift silently. Move them to a non-test
delimiter.go as the single source of truth.
Extract the alignment scaffolding so the format-specific PRs (text&code,
markdown golden, HTML) don't conflict on align_test.go.
Adds (test-only, no production behavior change):
- GoldenDoc{Meta,Items} and parseGolden: tolerant of both the legacy bare
array and the new {meta, items} golden format.
- LoadGoldenDoc, AcceptedDivergences(meta), FilterOutDocTypes(items, drop):
meta-driven divergence handling, no hardcoded divergence lists.
- TextCodeAlignOptions / DefaultTextCodeDelimiter, HTMLAlignOptions /
StripHTMLHeadingMarker: normalizer presets reused by the alignment tests.
- LoadGolden now tolerates the {meta, items} format.
Stacks on #18014 (the align_test.go framework is already in main).
Port dataset-level wiki incremental compile and refactor splitByTokens
token budgeting. Includes replace-only wiki merge, KNN dedup routing,
and template/config wiring.
## Summary
This PR improves the RAGFlow agentic-search path in three areas: it
stops the outer agent from re-looping over the same rag call, lets the
medium thinking mode discover and follow new sub-claims mid-loop, and
strengthens retrieval by having the LLM emit synonym-rich queries with
time/date/number terms boosted.
1. Avoid the outer re-loop — keep all multi-hop cycles inside agentic
RAG
2. Dynamic claims in medium mode — keep querying newly discovered
sub-questions
medium now enables allows_dynamic_claims. During orchestration, when
claim analysis discovers a new required sub-question
(discovered_claims), the loop spawns it as a new ClaimTarget and
continues searching it in subsequent cycles (bounded by the
dynamic-claim budget) instead of stopping. Also added:
3. Stronger query strategy — synonym-rich queries + time/date/number
weighting
LLM-generated synonyms: the claim-analysis prompt now instructs the
model to write each next_queries entry as a retrieval-boosted query that
actively folds in entity aliases, DATE/TIME synonyms (e.g. 1994 → 1994,
66th Academy Awards), and number/unit variants (e.g. 1.95 m → 6 ft 5
in).
Time/date/number boosting: query.py boosts numeric/date tokens to a high
weight (_NUM_DATE_TOKEN_RE).
