refa: optimize knowledge compilation concurrency (#16933)

### What problem does this PR solve?

Improve concurrency in the knowledge compilation pipeline:

- Run Compile LLM requests concurrently while preserving ordered
commits.
- Run merge flush tasks concurrently while keeping ES writes ordered.
- Improve concurrency for local deduplication, chain validation, and ES
deduplication.
- Remove temporary debugging instrumentation and unused timing
variables.

### Type of change

- [x] Refactor (no functional change)

rag/svr/task_executor_refactor/task_handler.py

@@ -730,6 +730,7 @@ class TaskHandler:
             "content_with_weight",
             "page_num_int",
             "top_int",
+            "compile_kwd",
         ]
         order_by = OrderByExpr()
         order_by.asc("page_num_int")
@@ -742,7 +743,15 @@ class TaskHandler:
                     settings.docStoreConn.search,
                     select_fields,
                     [],
-                    {"doc_id": [doc_id], "available_int": 1},
+                    {
+                        "doc_id": [doc_id],
+                        "available_int": 1,
+                        # Compilation writes its output back to the same
+                        # document index. Exclude those rows in the query so
+                        # they cannot change offset pagination while this
+                        # task is still streaming source chunks.
+                        "must_not": {"exists": "compile_kwd"},
+                    },
                     [],
                     order_by,
                     offset,