ragflow/rag/nlp/rag_tokenizer.py at 3285f09c92146fd09e28d9536b187f145b6c1821

mirror of https://github.com/infiniflow/ragflow.git synced 2026-09-08 10:14:35 +08:00

Files

T

qinling0210 2ffe6f7439 Import rag_tokenizer from Infinity (#11647 )

### What problem does this PR solve?

- Original rag/nlp/rag_tokenizer.py is put to Infinity and infinity-sdk
via https://github.com/infiniflow/infinity/pull/3117 .
Import rag_tokenizer from infinity and inherit from
rag_tokenizer.RagTokenizer in new rag/nlp/rag_tokenizer.py.

- Bump infinity to 0.6.8

### Type of change
- [x] Refactoring

2025-12-02 14:59:37 +08:00

1.3 KiB

Raw Blame History

View Raw

1.3 KiB Raw Blame History

1.3 KiB

Raw Blame History