feat: add LLM-guided semantic rechunking for knowledge compilation (#17546)

This commit is contained in:
buua436
2026-07-31 13:33:30 +08:00
committed by GitHub
parent 0969c04eca
commit 44e13d1cb6
20 changed files with 827 additions and 117 deletions

View File

@@ -1900,6 +1900,12 @@ Example: Virtual Hosted Style`,
rechunkByTreeLeaves: 'Re-chunk by tree leaves',
rechunkByTreeLeavesTip:
"Merge each leaf cluster's source chunks into a single replacement chunk. Originals are kept but marked unavailable for retrieval. Only one tree template per group may enable this.",
rechunkInput: 'Re-chunk parser output',
rechunkInputTip:
'Let the LLM determine chunk boundaries based on the knowledge compilation task.',
rechunkRules: 'Rechunking rules',
rechunkRulesPlaceholder:
'Describe how the LLM should group source chunks for this compilation task.',
jsonPreview: 'JSON preview',
processFlow: 'Process flow',
processFlowComingSoon: 'Process flow preview coming soon',

View File

@@ -1590,6 +1590,11 @@ NER使用 spaCy NER 和基于规则的关键词提取来抽取实体和关系
rechunkByTreeLeaves: '按树叶重新分块',
rechunkByTreeLeavesTip:
'将每个叶簇的源数据块合并为单个替换数据块。原始数据块保留但标记为不可检索。每个分组最多只能有一个树模板启用此功能。',
rechunkInput: '重新切分 Parser 输出',
rechunkInputTip: '根据知识编译任务,由 LLM 决定 chunk 边界。',
rechunkRules: 'Rechunk 规则',
rechunkRulesPlaceholder:
'描述此知识编译任务下LLM 应如何合并和划分源 chunk。',
jsonPreview: 'JSON 预览',
processFlow: '流程视图',
processFlowComingSoon: '流程视图预览即将到来',