mirror of
https://github.com/thedotmack/claude-mem.git
synced 2026-09-20 04:23:02 +08:00
Nguyen Thanh Dat
bfe4694498
* fix(search): answer CJK and Japanese queries by substring FTS5's unicode61 tokenizer has no delimiter to split Chinese or Japanese on, so a whole run of ideographs folds into a single token. Every query in those scripts is a substring of some longer run, and a substring of a token can never match it -- so observation and session search returned 0 rows for any multi-character CJK keyword, however many rows contained it. Route queries containing an unsegmented script to a substring predicate over the same columns the FTS table indexes. searchUserPrompts has always answered this way, escaping included, so the semantics already exist in this class; this extends them to the two tables that go through FTS. Queries FTS5 can tokenize are untouched. Fixes #3801 * fix(search): widen the unsegmented-script set to Bopomofo and Hangul Review on #3810 pointed out two scripts sitting beside the ones this branch already covered. Confirmed against tokenize='unicode61' rather than taken on faith: 설정 inside 설정을 -> 0 rows 설정을 as a whole token -> 1 row ㄓㄨ inside ㄓㄨㄛ -> 0 rows 项目 inside 修改了项目配置 -> 0 rows (the reported case) project inside "the project" -> 1 row (Latin, unaffected) unicode61 breaks on Unicode whitespace and punctuation, and none of these scripts put either between the characters of a run, so a run is one token and no substring of it can match. Korean is the interesting one: it does space its words, so only the sub-word case is affected. That case is still every partial-word query a user types, and the measurement above is exactly it. The blocks are adjacent, so \u3100-\u318F picks up Bopomofo together with the Hangul compatibility jamo beside it, and \uAC00-\uD7AF covers the syllables. Two tests, one per script. Narrowing the class back to the previous ranges fails both.