refactor(knowledge_compile): global compile pool, token-budget batching, and DocEngine-only deletion (#17679)

## Summary

This PR refactors the Go knowledge-compilation ingestion pipeline
(`internal/ingestion/knowledge_compile` +
`internal/ingestion/component/knowledge_compiler`) with three related
changes:

- **Token-budget batching for LLM merge decisions.**
`LLMMergeDecider.DecideBatch` previously stuffed every `(existing,
candidate)` pair into a single LLM call, risking `max_token` overflow.
It now splits pairs into token-bounded sub-batches (budget =
`llmMaxTokens * 0.85`) via `tokenizer.NumTokensFromString`, runs them
concurrently while preserving the global pair index, and never
reindexes.
- **Process-level global compile pool.** Introduces a single vCPU-sized
goroutine pool (`pool.go`, env `KC_COMPILE_CONCURRENCY`) dedicated to
*all* knowledge-compilation stages. KNN search loop, `DecideBatch`
sub-batches, `WriteMerged`/`DeleteMerged` internals, and the
component-level (structure/mindmap) per-call pools are all unified into
it via an injected submitter. No more per-job short-lived goroutines in
`runCompilerJobs` (futures are collected then awaited on the caller).
Fan-out stays bounded by the pool worker count; these stages are
docengine-bounded / LLM-bounded, not CPU-bounded.
- **DocEngine-only deletion.** `Consumer.processBatch` deletion no
longer loads the deleted docs' products into memory. Two sequential
DocEngine calls replace the old in-memory surgery:
- `DeleteDocLevelForDocs` — one `DeleteChunks` over `doc_id IN
deletedDocIDs` (merged rows carry `doc_id == kb`, so only per-doc
products match).
- `StripMergedSources` — one `Search` of `kc_merged=1` rows filtered by
`source_doc_ids IN deletedDocIDs` (intersection pushed down to the
engine), `UpdateChunks` the source array of survivors, and
`DeleteChunks` the rows whose array became empty.

## Changes

- `internal/ingestion/knowledge_compile/pool.go` (new): global
`compilerPool` +
`runCompilerJobs`/`SubmitCompilerJob`/`SubmitCompilerJobs`.
- `internal/ingestion/knowledge_compile/consumer.go`: deletion rewritten
to the two DocEngine calls;
`mergedBase`/`toDelete`/`stripDeletedSources` removed.
- `internal/ingestion/knowledge_compile/writer.go`:
`DeleteDocLevelForDocs` + `StripMergedSources` replace
`DeleteMergedForDoc`/`DeleteMerged`.
- `internal/ingestion/knowledge_compile/reader.go`: drop
`LoadMergedBySourceDoc` + `containsString` (keep `LoadDocProducts` for
the completion branch).
- `internal/ingestion/knowledge_compile/dedup.go`: `NewLLMDeduper` takes
`llmMaxTokens`; wires `SetMaxBatchTokens`/`SetSubmitter`.
- `internal/ingestion/knowledge_compiler/{structure,merge}.go`,
`mindmap/mindmap.go`, `pool_wiring.go`: token-budget split + submitter
injection.
- Tests: `structure_test.go` (token-budget split), `dedup_test.go`,
`consumer_test.go` (tombstone + DocEngine deletion assertions) updated.

## Validation

`bash build.sh --test -race ./internal/ingestion/knowledge_compile/...
./internal/ingestion/component/knowledge_compiler/...` passes (unit
tier, no external services).

🤖 Generated with [CodeBuddy](https://www.codebuddy.ai)

---------

Co-authored-by: yuzhichang <yuzhichang@infiniflow.ai>

internal/entity/knowledge_compile_doc.go

@@ -17,7 +17,7 @@ package entity
 
 import "time"
 
-// KnowledgeCompileDoc is the MySQL scheduling row for the dataset-level
+// KnowledgeCompileDataset is the MySQL scheduling row for the dataset-level
 // post-processing consumer (knowledge_compile_design.md §11.4, Option E). It is
 // the scheduling system of record: backlog_doc_ids holds the not-yet-processed
 // doc entries for the KB, inflight_doc_ids the ones a worker has claimed (the
@@ -29,7 +29,7 @@ import "time"
 // (doc_id + event_type + seq) as TEXT so the consumer can re-apply the same
 // out-of-order / tombstone guards as the broker-based design without re-reading
 // the queue.
-type KnowledgeCompileDoc struct {
+type KnowledgeCompileDataset struct {
 	DatasetID string `gorm:"primaryKey;column:dataset_id;size:64" json:"dataset_id"`
 	TenantID  string `gorm:"column:tenant_id;size:64;not null;default:''" json:"tenant_id"`
 	// The *_doc_ids columns store a JSON array as TEXT. No DDL default is set:
@@ -47,4 +47,4 @@ type KnowledgeCompileDoc struct {
 }
 
 // TableName pins the scheduling table name.
-func (KnowledgeCompileDoc) TableName() string { return "knowledge_compile_docs" }
+func (KnowledgeCompileDataset) TableName() string { return "knowledge_compile_docs" }