feat: add LLM-guided semantic rechunking for knowledge compilation (#17546)

api/db/init_data/compilation_templates/knowledge_graph.yaml

@@ -2,6 +2,12 @@ kind: knowledge_graph
 display_name: Knowledge graph
 config:
   kind: knowledge_graph
+  rechunk_rules: |
+    - Group only adjacent source chunks into coherent semantic units for this knowledge-compilation task.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Split when the topic, section, or semantic focus changes; do not merge unrelated content.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal structure.
+    - Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a robust graph entity extractor for knowledge graphs.

api/db/init_data/compilation_templates/mind_map.yaml

@@ -2,6 +2,12 @@ kind: mind_map
 display_name: Mind map - Radial concept hierarchy
 config:
   kind: mind_map
+  rechunk_rules: |
+    - Group only adjacent source chunks into coherent semantic units for this knowledge-compilation task.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Keep each main idea with its supporting content, and split when the topic or branch changes.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal structure.
+    - Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a robust mind-map extractor. Extract the central concept,

api/db/init_data/compilation_templates/page_index.yaml

@@ -2,6 +2,14 @@ kind: page_index
 display_name: PageIndex — Hierarchical table of contents
 config:
   kind: page_index
+  rechunk_rules: |
+    - Group only adjacent source chunks into coherent semantic units for this page-index task.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Keep each heading with the content it introduces, and split when the topic, section, or semantic focus changes.
+    - Preserve document heading hierarchy: keep a parent heading with its introductory content and nearby child-section context.
+    - Do not combine separate sections merely because their subjects are related; keep each section boundary recoverable.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal rows or items.
+    - Do not merge unrelated content. Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a source-grounded document index extractor. First extract every

api/db/init_data/compilation_templates/session_essence.yaml

@@ -2,6 +2,12 @@ kind: session_essence
 display_name: Session Essence — Cross-source entity synopses from conversations
 config:
   kind: knowledge_graph
+  rechunk_rules: |
+    - Group only adjacent source chunks that discuss the same conversation topic or entity.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Split when the speaker, topic, intent, or semantic focus changes.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal structure.
+    - Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a robust entity and fact extractor for conversational data.

api/db/init_data/compilation_templates/session_graph.yaml

@@ -2,6 +2,12 @@ kind: session_graph
 display_name: Session Graph — Knowledge graph from conversations
 config:
   kind: knowledge_graph
+  rechunk_rules: |
+    - Group only adjacent source chunks that discuss the same conversation topic or entity.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Split when the speaker, topic, intent, or semantic focus changes.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal structure.
+    - Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a robust entity and fact extractor for conversational data.

api/db/init_data/compilation_templates/timeline.yaml

@@ -2,6 +2,12 @@ kind: timeline
 display_name: List (Timeline) — Chronological events / Graph
 config:
   kind: timeline
+  rechunk_rules: |
+    - Group only adjacent source chunks that belong to the same event or continuous time period.
+    - Preserve source order: use every source chunk exactly once; do not omit, overlap, duplicate, or reorder chunks.
+    - Split when the event, time period, or semantic focus changes.
+    - Keep self-contained tables, lists, figures, and code blocks intact; do not split their internal structure.
+    - Do not invent, rewrite, or return source text; return only chunk grouping metadata.
   entity:
     description: >-
       You are a robust events-timeline extractor.