mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-09-08 10:14:35 +08:00
feat: add native Dutch language support for BM25 tokenization (#14140)
## Summary - Add language-aware Snowball stemmer to `RagTokenizer` supporting 16 languages (Dutch, German, French, Spanish, etc.) - Thread the KB `language` parameter through the full tokenization pipeline (14 parser modules + task executor) - Add Dutch to the frontend language lists and cross-language form ## Problem RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all** BM25 tokenization, regardless of the knowledge base language setting. This produces incorrect stems for non-English text. For example: | Dutch word | Dutch stemmer | English Porter | |---|---|---| | documenten | document | documenten (unchanged!) | | gebruikers | gebruiker | gebruik (over-stemmed) | | instellingen | instell | instellingen (unchanged!) | This degrades BM25 recall for any non-English knowledge base. ## Solution NLTK already ships Snowball stemmers for 16 languages. This PR: 1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with `set_language()` and `_normalize_token()` that selects the correct NLTK Snowball stemmer. Falls back to Porter for unmapped languages (Chinese, Japanese, Korean, etc. — these use character-based tokenization anyway). 2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** + **`rag/svr/task_executor.py`**: Threads the `language` parameter through `tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers. 3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`, `LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`, cross-language form field, and `en.ts` locale. ## Backward Compatibility - Default language is `"English"`, preserving existing behavior for all current users - Languages without a Snowball stemmer mapping fall back to Porter (no change) - No new dependencies — NLTK Snowball is already bundled
This commit is contained in:
+2
-2
@@ -548,7 +548,7 @@ def chunk(filename, binary=None, from_page=0, to_page=MAXIMUM_TASK_PAGE_NUMBER,
|
||||
else:
|
||||
d.update(stored)
|
||||
formatted_text = "\n".join([f"- {field}: {value}" for field, value in text_fields]) if text_fields else ""
|
||||
tokenize(d, formatted_text, eng)
|
||||
tokenize(d, formatted_text, eng, language=lang)
|
||||
if _debug_row_idx == 1:
|
||||
logger.debug(f"[TABLE_PARSER_DEBUG] Chunk content_with_weight length: {len(d.get('content_with_weight', '') or '')}")
|
||||
_cd = d.get("chunk_data")
|
||||
@@ -559,7 +559,7 @@ def chunk(filename, binary=None, from_page=0, to_page=MAXIMUM_TASK_PAGE_NUMBER,
|
||||
res.append(d)
|
||||
if tbls:
|
||||
doc = {"docnm_kwd": filename, "title_tks": rag_tokenizer.tokenize(re.sub(r"\.[a-zA-Z]+$", "", filename))}
|
||||
res.extend(tokenize_table(tbls, doc, is_english))
|
||||
res.extend(tokenize_table(tbls, doc, is_english, language=lang))
|
||||
callback(0.35, "")
|
||||
|
||||
return res
|
||||
|
||||
Reference in New Issue
Block a user