mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-14 08:58:27 +08:00
feat: add native Dutch language support for BM25 tokenization (#14140)
## Summary - Add language-aware Snowball stemmer to `RagTokenizer` supporting 16 languages (Dutch, German, French, Spanish, etc.) - Thread the KB `language` parameter through the full tokenization pipeline (14 parser modules + task executor) - Add Dutch to the frontend language lists and cross-language form ## Problem RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all** BM25 tokenization, regardless of the knowledge base language setting. This produces incorrect stems for non-English text. For example: | Dutch word | Dutch stemmer | English Porter | |---|---|---| | documenten | document | documenten (unchanged!) | | gebruikers | gebruiker | gebruik (over-stemmed) | | instellingen | instell | instellingen (unchanged!) | This degrades BM25 recall for any non-English knowledge base. ## Solution NLTK already ships Snowball stemmers for 16 languages. This PR: 1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with `set_language()` and `_normalize_token()` that selects the correct NLTK Snowball stemmer. Falls back to Porter for unmapped languages (Chinese, Japanese, Korean, etc. — these use character-based tokenization anyway). 2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** + **`rag/svr/task_executor.py`**: Threads the `language` parameter through `tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers. 3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`, `LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`, cross-language form field, and `en.ts` locale. ## Backward Compatibility - Default language is `"English"`, preserving existing behavior for all current users - Languages without a Snowball stemmer mapping fall back to Porter (no change) - No new dependencies — NLTK Snowball is already bundled
This commit is contained in:
@@ -22,6 +22,7 @@ const Languages = [
|
||||
'Vietnamese',
|
||||
'Arabic',
|
||||
'Turkish',
|
||||
'Dutch',
|
||||
];
|
||||
|
||||
export const crossLanguageOptions = Languages.map((x) => ({
|
||||
|
||||
@@ -58,6 +58,7 @@ export const LanguageList = [
|
||||
'Bulgarian',
|
||||
'Arabic',
|
||||
'Turkish',
|
||||
'Dutch',
|
||||
];
|
||||
export const LanguageMap = {
|
||||
English: 'English',
|
||||
@@ -76,6 +77,7 @@ export const LanguageMap = {
|
||||
Bulgarian: 'Български',
|
||||
Arabic: 'العربية',
|
||||
Turkish: 'Türkçe',
|
||||
Dutch: 'Nederlands',
|
||||
};
|
||||
|
||||
export enum LanguageAbbreviation {
|
||||
@@ -95,6 +97,7 @@ export enum LanguageAbbreviation {
|
||||
Ar = 'ar',
|
||||
Tr = 'tr',
|
||||
Ko = 'ko',
|
||||
Nl = 'nl',
|
||||
}
|
||||
|
||||
export const LanguageAbbreviationMap = {
|
||||
@@ -114,6 +117,7 @@ export const LanguageAbbreviationMap = {
|
||||
[LanguageAbbreviation.Ar]: 'العربية',
|
||||
[LanguageAbbreviation.Tr]: 'Türkçe',
|
||||
[LanguageAbbreviation.Ko]: '한국어',
|
||||
[LanguageAbbreviation.Nl]: 'Nederlands',
|
||||
};
|
||||
|
||||
export const LanguageTranslationMap = {
|
||||
@@ -143,6 +147,7 @@ export const LanguageTranslationMap = {
|
||||
Bulgarian: 'bg',
|
||||
Arabic: 'ar',
|
||||
Turkish: 'tr',
|
||||
Dutch: 'nl',
|
||||
};
|
||||
|
||||
export enum FileMimeType {
|
||||
|
||||
@@ -40,6 +40,7 @@ export default {
|
||||
bulgarian: 'Bulgarian',
|
||||
arabic: 'Arabic',
|
||||
turkish: 'Turkish',
|
||||
dutch: 'Dutch',
|
||||
language: 'Language',
|
||||
languageMessage: 'Please input your language!',
|
||||
languagePlaceholder: 'select your language',
|
||||
@@ -3064,6 +3065,7 @@ Important structured information may include: names, dates, locations, events, k
|
||||
bulgarian: 'Bulgarian',
|
||||
arabic: 'Arabic',
|
||||
turkish: 'Turkish',
|
||||
dutch: 'Dutch',
|
||||
},
|
||||
pagination: {
|
||||
total: 'Total {{total}}',
|
||||
|
||||
Reference in New Issue
Block a user