feat: add native Dutch language support for BM25 tokenization (#14140)

## Summary
- Add language-aware Snowball stemmer to `RagTokenizer` supporting 16
languages (Dutch, German, French, Spanish, etc.)
- Thread the KB `language` parameter through the full tokenization
pipeline (14 parser modules + task executor)
- Add Dutch to the frontend language lists and cross-language form

## Problem
RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all**
BM25 tokenization, regardless of the knowledge base language setting.
This produces incorrect stems for non-English text. For example:

| Dutch word | Dutch stemmer | English Porter |
|---|---|---|
| documenten | document | documenten (unchanged!) |
| gebruikers | gebruiker | gebruik (over-stemmed) |
| instellingen | instell | instellingen (unchanged!) |

This degrades BM25 recall for any non-English knowledge base.

## Solution
NLTK already ships Snowball stemmers for 16 languages. This PR:

1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with
`set_language()` and `_normalize_token()` that selects the correct NLTK
Snowball stemmer. Falls back to Porter for unmapped languages (Chinese,
Japanese, Korean, etc. — these use character-based tokenization anyway).
2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** +
**`rag/svr/task_executor.py`**: Threads the `language` parameter through
`tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers.
3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`,
`LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`,
cross-language form field, and `en.ts` locale.

## Backward Compatibility
- Default language is `"English"`, preserving existing behavior for all
current users
- Languages without a Snowball stemmer mapping fall back to Porter (no
change)
- No new dependencies — NLTK Snowball is already bundled
This commit is contained in:
Rodger Blom
2026-07-06 17:39:56 +02:00
committed by GitHub
parent dd20561fca
commit d8cefcf052
20 changed files with 73 additions and 53 deletions

View File

@@ -22,6 +22,7 @@ const Languages = [
'Vietnamese',
'Arabic',
'Turkish',
'Dutch',
];
export const crossLanguageOptions = Languages.map((x) => ({

View File

@@ -58,6 +58,7 @@ export const LanguageList = [
'Bulgarian',
'Arabic',
'Turkish',
'Dutch',
];
export const LanguageMap = {
English: 'English',
@@ -76,6 +77,7 @@ export const LanguageMap = {
Bulgarian: 'Български',
Arabic: 'العربية',
Turkish: 'Türkçe',
Dutch: 'Nederlands',
};
export enum LanguageAbbreviation {
@@ -95,6 +97,7 @@ export enum LanguageAbbreviation {
Ar = 'ar',
Tr = 'tr',
Ko = 'ko',
Nl = 'nl',
}
export const LanguageAbbreviationMap = {
@@ -114,6 +117,7 @@ export const LanguageAbbreviationMap = {
[LanguageAbbreviation.Ar]: 'العربية',
[LanguageAbbreviation.Tr]: 'Türkçe',
[LanguageAbbreviation.Ko]: '한국어',
[LanguageAbbreviation.Nl]: 'Nederlands',
};
export const LanguageTranslationMap = {
@@ -143,6 +147,7 @@ export const LanguageTranslationMap = {
Bulgarian: 'bg',
Arabic: 'ar',
Turkish: 'tr',
Dutch: 'nl',
};
export enum FileMimeType {

View File

@@ -40,6 +40,7 @@ export default {
bulgarian: 'Bulgarian',
arabic: 'Arabic',
turkish: 'Turkish',
dutch: 'Dutch',
language: 'Language',
languageMessage: 'Please input your language!',
languagePlaceholder: 'select your language',
@@ -3064,6 +3065,7 @@ Important structured information may include: names, dates, locations, events, k
bulgarian: 'Bulgarian',
arabic: 'Arabic',
turkish: 'Turkish',
dutch: 'Dutch',
},
pagination: {
total: 'Total {{total}}',