mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-15 20:27:20 +08:00
d8cefcf052
## Summary - Add language-aware Snowball stemmer to `RagTokenizer` supporting 16 languages (Dutch, German, French, Spanish, etc.) - Thread the KB `language` parameter through the full tokenization pipeline (14 parser modules + task executor) - Add Dutch to the frontend language lists and cross-language form ## Problem RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all** BM25 tokenization, regardless of the knowledge base language setting. This produces incorrect stems for non-English text. For example: | Dutch word | Dutch stemmer | English Porter | |---|---|---| | documenten | document | documenten (unchanged!) | | gebruikers | gebruiker | gebruik (over-stemmed) | | instellingen | instell | instellingen (unchanged!) | This degrades BM25 recall for any non-English knowledge base. ## Solution NLTK already ships Snowball stemmers for 16 languages. This PR: 1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with `set_language()` and `_normalize_token()` that selects the correct NLTK Snowball stemmer. Falls back to Porter for unmapped languages (Chinese, Japanese, Korean, etc. — these use character-based tokenization anyway). 2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** + **`rag/svr/task_executor.py`**: Threads the `language` parameter through `tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers. 3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`, `LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`, cross-language form field, and `en.ts` locale. ## Backward Compatibility - Default language is `"English"`, preserving existing behavior for all current users - Languages without a Snowball stemmer mapping fall back to Porter (no change) - No new dependencies — NLTK Snowball is already bundled
79 lines
1.8 KiB
TypeScript
79 lines
1.8 KiB
TypeScript
import {
|
|
FormControl,
|
|
FormField,
|
|
FormItem,
|
|
FormLabel,
|
|
} from '@/components/ui/form';
|
|
import { MultiSelect } from '@/components/ui/multi-select';
|
|
import { cn } from '@/lib/utils';
|
|
import { t } from 'i18next';
|
|
import { toLower } from 'lodash';
|
|
import { useFormContext } from 'react-hook-form';
|
|
import { useTranslation } from 'react-i18next';
|
|
|
|
const Languages = [
|
|
'English',
|
|
'Chinese',
|
|
'Spanish',
|
|
'French',
|
|
'German',
|
|
'Japanese',
|
|
'Korean',
|
|
'Vietnamese',
|
|
'Arabic',
|
|
'Turkish',
|
|
'Dutch',
|
|
];
|
|
|
|
export const crossLanguageOptions = Languages.map((x) => ({
|
|
label: t('language.' + toLower(x)),
|
|
value: x,
|
|
}));
|
|
|
|
type CrossLanguageItemProps = {
|
|
name?: string;
|
|
vertical?: boolean;
|
|
label?: string;
|
|
};
|
|
|
|
export const CrossLanguageFormField = ({
|
|
name = 'prompt_config.cross_languages',
|
|
vertical = true,
|
|
label,
|
|
}: CrossLanguageItemProps) => {
|
|
const { t } = useTranslation();
|
|
const form = useFormContext();
|
|
|
|
return (
|
|
<FormField
|
|
control={form.control}
|
|
name={name}
|
|
render={({ field }) => (
|
|
<FormItem
|
|
className={cn('flex', {
|
|
'gap-2': vertical,
|
|
'flex-col': vertical,
|
|
'justify-between': !vertical,
|
|
'items-center': !vertical,
|
|
})}
|
|
>
|
|
<FormLabel tooltip={t('chat.crossLanguageTip')}>
|
|
{label || t('chat.crossLanguage')}
|
|
</FormLabel>
|
|
<FormControl>
|
|
<MultiSelect
|
|
options={crossLanguageOptions}
|
|
placeholder={t('chat.crossLanguagePlaceholder')}
|
|
maxCount={100}
|
|
{...field}
|
|
onValueChange={field.onChange}
|
|
defaultValue={field.value}
|
|
modalPopover
|
|
/>
|
|
</FormControl>
|
|
</FormItem>
|
|
)}
|
|
/>
|
|
);
|
|
};
|