- Rename the English sample/golden to *_en.* and add a Chinese
counterpart (*_zh.*) so markdown parsing is exercised in both Latin
and CJK contexts; the Chinese sample also covers the full-width
delimiters in the default delimiter set (\n!?;。；！？).
- Both baselines keep the {meta, items} form with accepted_divergences
declared in meta; the test is now table-driven over en/zh.
- The generator script is not committed; regeneration is described in
each golden's meta block.
Stacks on the markdown golden meta-driven refactor (c84236ace).
Move the markdown golden baseline to the {meta, items} format and drive
accepted divergences from meta.accepted_divergences instead of hardcoded lists.
No production parser code changes.
- markdown.python.golden.json is now {meta, items}; meta records how it was
produced so it is reproducible without a committed script.
- TestMarkdownParser_AlignmentGolden uses LoadGoldenDoc / AcceptedDivergences /
FilterOutDocTypes (from the foundation helpers PR).
- Deletes testdata/gen_markdown_golden.py.
Depends on #18014 and the foundation helpers PR (align_test.go).
Files:
internal/parser/parser/markdown_parser_test.go
internal/parser/parser/testdata/markdown.python.golden.json
internal/parser/parser/testdata/markdown.sample.md
internal/parser/parser/testdata/gen_markdown_golden.py (deleted)
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.
Port dataset-level wiki incremental compile and refactor splitByTokens
token budgeting. Includes replace-only wiki merge, KNN dedup routing,
and template/config wiring.
## Summary
This PR improves the RAGFlow agentic-search path in three areas: it
stops the outer agent from re-looping over the same rag call, lets the
medium thinking mode discover and follow new sub-claims mid-loop, and
strengthens retrieval by having the LLM emit synonym-rich queries with
time/date/number terms boosted.
1. Avoid the outer re-loop — keep all multi-hop cycles inside agentic
RAG
2. Dynamic claims in medium mode — keep querying newly discovered
sub-questions
medium now enables allows_dynamic_claims. During orchestration, when
claim analysis discovers a new required sub-question
(discovered_claims), the loop spawns it as a new ClaimTarget and
continues searching it in subsequent cycles (bounded by the
dynamic-claim budget) instead of stopping. Also added:
3. Stronger query strategy — synonym-rich queries + time/date/number
weighting
LLM-generated synonyms: the claim-analysis prompt now instructs the
model to write each next_queries entry as a retrieval-boosted query that
actively folds in entity aliases, DATE/TIME synonyms (e.g. 1994 → 1994,
66th Academy Awards), and number/unit variants (e.g. 1.95 m → 6 ft 5
in).
Time/date/number boosting: query.py boosts numeric/date tokens to a high
weight (_NUM_DATE_TOKEN_RE).
