Feat: Add knowledge compilation workflows (#16515)

## Summary
- Add knowledge compilation template APIs, services, and builtin
template seed data
- Add advanced knowledge compile structure/artifact/RAPTOR workflow
support
- Update parsing, dataset/document APIs, and supporting services for
compilation workflows

api/db/init_data/compilation_templates/artifacts.yaml

@@ -0,0 +1,136 @@
+kind: artifacts
+display_name: Artifacts — Graph-based wiki
+config:
+  kind: artifacts
+  example: |
+      - Each page must be a proper encyclopedic article, NOT a flat bullet list:
+      - 1. Opening paragraph (2-4 sentences defining what this is). No heading.
+      - 2. Sections with H2 headings, each starting with prose before sub-bullets.
+      - 3. Bold key terms on first use; link them with [[ ]] artifactlinks.
+      - 4. Examples or implications where the source provides them.
+      - 5. ## See also section at the end with artifactlinks to highly related pages(less than 12).\n
+      - Page structure could be as following: (Not provided)
+  entity:
+    description: >-
+      You are a robust graph entity extractor for knowledge graphs.
+    fields:
+      - type: person
+        description: A natural person (individual human).
+        rule: |
+          - Full name preferred (e.g., "Elon Musk", not "Musk" alone if ambiguous).
+          - Include titles only if integral to identity (e.g., "Dr. Smith").
+          - Max length: 60 characters.
+      - type: org
+        description: Organization, company, institution, agency, or any collective group.
+        rule: |
+          - Use the official name when possible (e.g., "United Nations").
+          - Abbreviations accepted if widely known (e.g., "UN", "NASA").
+          - Max length: 80 characters.
+      - type: product
+        description: Tangible or intangible product, service, software, or offering.
+        rule: |
+          - Include version numbers if relevant (e.g., "iPhone 14").
+          - Generic categories (e.g., "smartphone") only if no specific name is given.
+          - Max length: 100 characters.
+      - type: regulation
+        description: Law, policy, standard, guideline, or regulatory document.
+        rule: |
+          - Use official title or identifier (e.g., "GDPR", "OSHA standard 1910").
+          - Include jurisdiction if known (e.g., "EU GDPR").
+          - Max length: 120 characters.
+      - type: location
+        description: Geographic place (country, city, address, region, natural feature).
+        rule: |
+          - Hierarchical format allowed (e.g., "Paris, France").
+          - Avoid overly vague terms (e.g., "there") unless resolved.
+          - Max length: 80 characters.
+      - type: system
+        description: Technical system, platform, framework, or infrastructure.
+        rule: |
+          - Distinct from product: system implies integrated environment
+            (e.g., "Linux OS", "power grid").
+          - Use proper naming.
+          - Max length: 100 characters.
+      - type: equipment
+        description: Physical device, machinery, hardware, or tool.
+        rule: |
+          - Specific model preferred (e.g., "Boeing 737").
+          - Generic allowed only if precise type (e.g., "drill press").
+          - Max length: 80 characters.
+      - type: other
+        description: Entities that do not fit any above category.
+        rule: |
+          - Use sparingly; prefer mapping to a defined type when possible.
+          - Still provide a meaningful label.
+          - Max length: 80 characters.
+  relation:
+    description: >-
+      You are an expert in extracting semantic relations between entities.
+    fields:
+      - type: owns
+        description: Ownership or possession (legal or de facto).
+        rule: |
+          - Direction from owner to owned: (A owns B).
+          - Example: "Company A owns product B".
+      - type: part_of
+        description: Mereological relation — component to whole.
+        rule: |
+          - Direction from part to whole: (A part_of B).
+          - Example: "Engine part_of car".
+      - type: caused_by
+        description: Causal relation — event, action, or state leads to another.
+        rule: |
+          - Direction from effect to cause: (A caused_by B) meaning B causes A.
+          - Example: "Accident caused_by brake failure".
+      - type: regulates
+        description: Regulatory or governing relation (law/standard controls entity).
+        rule: |
+          - Direction from regulator to regulated: (A regulates B).
+          - Example: "GDPR regulates data processing".
+      - type: uses
+        description: Utilization — an entity employs or consumes another entity.
+        rule: |
+          - Direction from user to used: (A uses B).
+          - Example: "System uses equipment".
+      - type: located_in
+        description: Spatial containment — entity situated inside a location.
+        rule: |
+          - Direction from located entity to containing location: (A located_in B).
+          - Example: "Office located_in city".
+      - type: other
+        description: Any meaningful relation not covered by the above types.
+        rule: |
+          - Provide an explicit label in a "relation_label" field.
+          - Direction must be clear.
+  claim:
+    fields:
+      - statement: >-
+          A complete factual sentence stated in the source. Any sentence of the form
+          'X is Y', 'X has Y', 'X does Y', 'X was founded in Y', 'X is located in Y',
+          'X reported Y', etc. is a claim. Aim for at least 1-3 claims per entity per
+          chunk that mentions it.
+        subject: >-
+          Entity/concept this claim is about (must match one of the entity/concept
+          names extracted above).
+  concept:
+    fields:
+      - term: >-
+          Concept name OR a thematic section topic (prefer the source's heading
+          wording when coherent).
+        definition_excerpt: >-
+          Verbatim or near-verbatim defining phrase from the chunk.
+  global_rules: |
+    - Each relation links two entities (subject → object) with a predicate type.
+    - Format: {"subject_id": "<entity_id>", "predicate": "<type>",
+      "object_id": "<entity_id>", "chunk_id": "<chunk_ID>"}.
+    - Both subject and object must be previously extracted entities.
+    - If ambiguous direction, choose the most logical default
+      (e.g., "part_of" always from part to whole).
+    - When multiple relations appear in a chunk, list all in order of appearance.
+    - Keep language consistent; relation type is always English (the given type name).
+    - Every extracted entity must have exactly one type from the list.
+    - Entity label (the text representing the entity) is required, non-empty.
+    - Format: {"entity_id": "<unique_id>", "type": "<type>", "label": "<label>",
+      "chunk_id": "<chunk_ID>"}.
+    - If no entity in a chunk, output no entity for that chunk.
+    - Keep the chunks' original language (Chinese/English etc.) for entities and relations.

api/db/init_data/compilation_templates/empty.yaml

@@ -0,0 +1,17 @@
+kind: empty
+display_name: Empty
+config:
+  kind: empty
+  entity:
+    description: ''
+    fields:
+      - type: ''
+        description: ''
+        rule: ''
+  relation:
+    description: ''
+    fields:
+      - type: ''
+        description: ''
+        rule: ''
+  global_rules: ''

api/db/init_data/compilation_templates/knowledge_graph.yaml

@@ -0,0 +1,74 @@
+kind: knowledge_graph
+display_name: Knowledge graph
+config:
+  kind: knowledge_graph
+  entity:
+    description: >-
+      You are a robust graph entity extractor for knowledge graphs.
+    fields:
+      - type: person
+        description: A natural person (individual human).
+        rule: |
+          - Full name preferred (e.g., "Elon Musk", not "Musk" alone if ambiguous).
+          - Include titles only if integral to identity (e.g., "Dr. Smith").
+          - Max length: 60 characters.
+      - type: org
+        description: Organization, company, institution, agency, or any collective group.
+        rule: |
+          - Use the official name when possible (e.g., "United Nations").
+          - Abbreviations accepted if widely known (e.g., "UN", "NASA").
+          - Max length: 80 characters.
+      - type: product
+        description: Tangible or intangible product, service, software, or offering.
+        rule: |
+          - Include version numbers if relevant (e.g., "iPhone 14").
+          - Generic categories (e.g., "smartphone") only if no specific name is given.
+          - Max length: 100 characters.
+      - type: regulation
+        description: Law, policy, standard, guideline, or regulatory document.
+        rule: |
+          - Use official title or identifier (e.g., "GDPR", "OSHA standard 1910").
+          - Include jurisdiction if known (e.g., "EU GDPR").
+          - Max length: 120 characters.
+      - type: location
+        description: Geographic place (country, city, address, region, natural feature).
+        rule: |
+          - Hierarchical format allowed (e.g., "Paris, France").
+          - Avoid overly vague terms (e.g., "there") unless resolved.
+          - Max length: 80 characters.
+      - type: other
+        description: Entities that do not fit any above category.
+        rule: |
+          - Use sparingly; prefer mapping to a defined type when possible.
+          - Still provide a meaningful label.
+          - Max length: 80 characters.
+  relation:
+    description: >-
+      You are an expert in extracting semantic relations between entities.
+    fields:
+      - type: owns
+        description: Ownership or possession (legal or de facto).
+        rule: |
+          - Direction from owner to owned: (A owns B).
+      - type: part_of
+        description: Mereological relation — component to whole.
+        rule: |
+          - Direction from part to whole: (A part_of B).
+      - type: caused_by
+        description: Causal relation — event, action, or state leads to another.
+        rule: |
+          - Direction from effect to cause: (A caused_by B) meaning B causes A.
+      - type: regulates
+        description: Regulatory or governing relation (law/standard controls entity).
+        rule: |
+          - Direction from regulator to regulated: (A regulates B).
+      - type: located_in
+        description: Spatial containment — entity situated inside a location.
+        rule: |
+          - Direction from located entity to containing location: (A located_in B).
+      - type: other
+        description: Any meaningful relation not covered by the above types.
+        rule: |
+          - Provide an explicit label in a "relation_label" field.
+          - Direction must be clear.
+  global_rules: ''

api/db/init_data/compilation_templates/mind_map.yaml

@@ -0,0 +1,69 @@
+kind: mind_map
+display_name: Mind map - Radial concept hierarchy
+config:
+  kind: mind_map
+  entity:
+    description: >-
+      You are a robust mind-map extractor. Extract the central concept,
+      major branches, and supporting sub-branches needed to build a
+      non-linear visual hierarchy around the main subject.
+    fields:
+      - type: central_topic
+        description: The main idea, goal, problem, or subject placed at the center of the mind map.
+        rule: |
+          - Prefer a concise noun phrase that represents the whole chunk or document section.
+          - Use the source wording when it clearly names the topic.
+          - If no central topic can be inferred, output "-1".
+          - Max length: 80 characters.
+      - type: branch
+        description: A major theme, category, or dimension radiating from the central topic.
+        rule: |
+          - Use broad categories that organize multiple related details.
+          - Avoid duplicating the central topic as a branch.
+          - Keep labels concise and parallel when possible.
+          - Max length: 80 characters.
+      - type: sub_branch
+        description: A specific detail, task, supporting concept, example, or item under a branch.
+        rule: |
+          - Attach each sub-branch to the most specific relevant parent branch.
+          - Preserve important source terms, names, quantities, and constraints.
+          - Split unrelated details into separate sub-branches.
+          - Max length: 120 characters.
+      - type: keyword
+        description: A compact supporting word or phrase that clarifies a branch or sub-branch.
+        rule: |
+          - Use only meaningful terms that improve scanability.
+          - Do not include filler words or generic labels.
+          - Max length: 60 characters.
+  relation:
+    description: >-
+      You are an expert hierarchical reasoning assistant specializing in
+      mind-map structure. Link each concept to its parent so the result
+      forms an intuitive radial hierarchy.
+    fields:
+      - type: has_branch
+        description: The central topic contains a major branch.
+        rule: |
+          - Direction from central topic to branch: (A has_branch B).
+          - Use only for first-level branches radiating from the center.
+      - type: has_sub_branch
+        description: A branch or sub-branch contains a more specific child concept.
+        rule: |
+          - Direction from parent to child: (A has_sub_branch B).
+          - Use recursively for second-level and deeper details.
+      - type: supports
+        description: A keyword, example, task, or detail supports a parent concept.
+        rule: |
+          - Direction from supporting item to supported concept: (A supports B).
+          - Use when the item explains, evidences, or clarifies the parent.
+      - type: related_to
+        description: A cross-link between two concepts that are associated but not parent-child.
+        rule: |
+          - Use sparingly; prefer parent-child hierarchy when possible.
+          - Direction may follow the stronger explanatory dependency.
+  global_rules: |
+    - Build a mind map around one central concept whenever possible.
+    - Arrange major themes as first-level branches and details as recursive sub-branches.
+    - Keep labels short enough for visual nodes; avoid sentence-length node names.
+    - Preserve a clear hierarchy; do not create cycles in parent-child relations.
+    - Use the same language as the source text unless normalization is necessary.

api/db/init_data/compilation_templates/page_index.yaml

@@ -0,0 +1,33 @@
+kind: page_index
+display_name: PageIndex — Hierarchical table of contents
+config:
+  kind: page_index
+  entity:
+    description: >-
+      You are a robust Table-of-Contents (TOC) extractor.
+    fields:
+      - type: title
+        description: the heading text (clean, no page numbers or leader dots)
+        rule: |
+          - Length restriction:
+              • Chinese heading: ≤25 characters
+              • English heading: ≤80 characters
+          - "title" must be non-empty (or exactly "-1").
+          - If any part of a chunk has no valid heading, output that part as {"title":"-1", ...}.
+          - If a chunk contains multiple headings, expand them in order:
+              - Each heading → {"title":"...","chunk_id":"<chunk_ID>"}.
+          - When ambiguous, prefer "-1" unless the text strongly looks like a heading.
+          - Keep language of "title" the same as the input.
+          - Prefix like following must be titles: 第x章, 第N条, 第N节, 1, 1.1, 1.1.1 ...
+  relation:
+    description: >-
+      You are an expert logical reasoning assistant specializing in hierarchical titles.
+    fields:
+      - type: include
+        description: Upper-level title includes lower-level title.
+        rule: |
+          - "-1" is an invalid title; it does not belong to or include any other titles.
+          - Must follow the hierarchical index/numbering (e.g., "1", "2.1", "3.2.5") when present.
+          - Keep language of "title" the same as the input.
+          - 第N章 must include 第N条 or 第N节.
+  global_rules: ''

api/db/init_data/compilation_templates/timeline.yaml

@@ -0,0 +1,48 @@
+kind: timeline
+display_name: List (Timeline) — Chronological events / Graph
+config:
+  kind: timeline
+  entity:
+    description: >-
+      You are a robust events-timeline extractor.
+    fields:
+      - type: timestamp
+        description: the date or time reference (clean, no extraneous text)
+        rule: |
+          - Format: prefer ISO 8601 (YYYY-MM-DD) or a normalized human-readable form
+            (e.g., "March 5, 2024").
+          - If only a relative time (e.g., "yesterday", "next week"), convert to
+            absolute when the context allows, else keep as is.
+          - "timestamp" must be non-empty (or exactly "-1" if no valid time/date).
+          - If a chunk contains multiple events with distinct timestamps, expand them
+            in chronological order:
+              - Each event → {"timestamp":"...","event":"...","chunk_id":"<chunk_ID>"}.
+          - When ambiguous, prefer "-1" unless the text strongly indicates a specific
+            time/date.
+          - Keep language and numbering style of "timestamp" consistent with the input.
+      - type: event
+        description: the event description associated with the timestamp (concise, no metadata)
+        rule: |
+          - Length restriction:
+              • Chinese event: ≤40 characters
+              • English event: ≤120 characters
+          - "event" must be non-empty (or exactly "-1") if no valid event description.
+          - If no valid event but a timestamp exists, output
+            {"timestamp":"...","event":"-1", ...}.
+          - Preserve the core action and key entities; omit redundant phrasing.
+  relation:
+    description: >-
+      You are an expert sequential reasoning assistant specializing in chronological
+      timelines.
+    fields:
+      - type: ordered
+        description: Events are arranged in strict chronological order (earliest to latest).
+        rule: |
+          - "-1" for timestamp or event indicates invalid or missing data; such entries
+            do not participate in ordering.
+          - Must follow explicit or inferred temporal indicators
+            (e.g., "then", "afterwards", "at 3 PM").
+          - If multiple events share the same timestamp, preserve their textual order
+            as a sub-list.
+          - Keep language and date formatting consistent across the timeline.
+  global_rules: ''

api/db/init_data/compilation_templates/tree.yaml

@@ -0,0 +1,31 @@
+kind: tree
+display_name: Tree — RAPTOR-based document tree
+description: >-
+  Recursive Abstractive Processing for Tree-Organized Retrieval over a
+  single document's chunks. Produces a hierarchical summary tree
+  (root summary → cluster summaries → leaf cluster source_chunk_ids).
+config:
+  kind: tree
+  raptor:
+    prompt: |-
+      Please summarize the following paragraphs. Be careful with the numbers, do not make things up. Paragraphs as following:
+            {cluster_content}
+      The above is the content you need to summarize.
+    max_token: 512
+    threshold: 0.1
+  # Entity / relation collections are not used by the tree kind but kept
+  # as empty stubs so the form schema's shared validators see a stable
+  # shape regardless of kind.
+  entity:
+    description: ''
+    fields:
+      - type: ''
+        description: ''
+        rule: ''
+  relation:
+    description: ''
+    fields:
+      - type: ''
+        description: ''
+        rule: ''
+  global_rules: ''