### What problem does this PR solve?
Incremental Wiki compilation could lose provenance for claim-light
entities, produce unstable page groups across embedding models, route
entities to unrelated pages, and assign topics without sufficient
page-level context. Document removals and page membership changes could
also leave stale Wiki state.
This PR:
- preserves source document and chunk provenance throughout entity
matching, reduction, page generation, and deletion;
- uses embeddings to retrieve candidates and the LLM to make final page
grouping and incremental routing decisions;
- batches embedding and LLM operations with bounded concurrency and
deterministic fallbacks;
- selects source-scoped topic candidates with embeddings before the page
LLM chooses the final topic;
- rebuilds Wiki state when the compilation mode or embedding model
changes;
- normalizes Wiki array fields returned by the API and retains entities
without relations in graph responses.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
### What problem does this PR solve?
Fix Infinity compatibility issues in knowledge compilation.
This change:
- Stores compilation source ID lists as JSON arrays in Infinity.
- Parses JSON array fields when reading compiled documents.
- Uses `json_contains` for filtering JSON array fields.
- Adds the missing `name` column to the Infinity mapping.
- Updates dataset navigation KNN search to use the unified
`MatchDenseExpr` interface.
- Handles unavailable embeddings without querying an invalid `q_0_vec`
field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Improve concurrency in the knowledge compilation pipeline:
- Run Compile LLM requests concurrently while preserving ordered
commits.
- Run merge flush tasks concurrently while keeping ES writes ordered.
- Improve concurrency for local deduplication, chain validation, and ES
deduplication.
- Remove temporary debugging instrumentation and unused timing
variables.
### Type of change
- [x] Refactor (no functional change)
