refactor: overhaul task executor with layered architecture and comprehensive test suite (#15471)

## Summary

Decomposes the monolithic `task_executor.py` (1945 lines) into a 6-layer
architecture with clear separation of concerns. The refactored code is
functionally equivalent to the original, verified through 400 passing
tests and a production-vs-dry-run comparison framework.

## Architecture

```
entry (task_manager)
  └─ orchestration (task_handler)
       ├─ services (chunk_service, embedding_service, dataflow_service, raptor_service, post_processor)
       │    └─ utilities (chunk_builder, chunk_post_processor, embedding_utils)
       └─ infrastructure (task_context, recording_context, interceptor)
```

Key design decisions:
- **TaskContext** — typed facade over raw task dict, injects rate
limiters + callbacks via composition
- **RecordingContext + Comparator** — enables side-by-side production vs
dry-run execution for safe migration
- **NullRecordingContext** — zero-allocation no-op for production, uses
`__slots__`
- **WriteOperationInterceptor** — FIFO replay of previous runs function
returns for comparison mode

## Migration Strategy

The original `handle_task()` in `task_executor.py` uses a 3-way switch
via `TE_RUN_MODE`:
- `TE_RUN_MODE=0` (default) → runs refactored code
- `TE_RUN_MODE=1` → runs both original + refactored, compares all
intermediate results
- `TE_RUN_MODE=2` → runs original code (fallback)

The comparison mode (`TE_RUN_MODE=1`) records ~40 intermediate values
(chunks, vectors, token counts, func return values) from the production
run and replays them during dry-run, then uses `ContextComparator` to
report mismatches.

## Functional Equivalence Fixes

All divergences between original and refactored code were identified and
fixed:
- Timeout decorators (handle/build_chunks/raptor/embedding)
- NullRecordingContext leak in finally block causing RuntimeError
- MinIO None-binary check with proper FileNotFoundError
- Dataflow dispatch after embedding binding + init_kb
- Memory task missing return after processing
- RAPTOR checkpoint progress reporting
- Tag cache (get_tags_from_cache/set_tags_to_cache) restoration
- dataflow_id correction in _load_dsl
- Language default Chinese, dead code guard removal
- embed_chunks made async with proper thread_pool_exec
- Full GraphRAG default configuration (10 parameters)
- Hardcoded q_768_vec fallback removal in RAPTOR

## Test Changes

- 20 new tests covering table parser manual mode, tag cache, embedding
edge cases, RAPTOR checkpoint, dataflow_id correction, storage binary
None, cancel cleanup, metadata=None boundary
- Unified `make_task_context`/`make_task_dict` factories eliminated 10+
duplicated helpers
- DataflowService tests migrated from internal method mocks to IO
boundary mocks (real orchestration code executes)
- Parametrized duplicate build_chunks post-processor tests
- 7 raptor tests modernized to @pytest.mark.asyncio
- Mock count per test reduced through boundary-level mocking strategy

**Test count: 400 passing, 0 warnings, 0 skips**

## Files Changed

| File | Change |
|------|--------|
| `rag/svr/task_executor.py` | +1 line (NullRecordingContext fix) |
| `rag/svr/task_executor_refactor/task_handler.py` | Orchestration
layer, 8 logic fixes |
| `rag/svr/task_executor_refactor/chunk_service.py` | +timeout +
None-check |
| `rag/svr/task_executor_refactor/embedding_service.py` | sync→async
rewrite |
| `rag/svr/task_executor_refactor/dataflow_service.py` | dataflow_id fix
+ timeout |
| `rag/svr/task_executor_refactor/raptor_service.py` | checkpoint fix +
assert |
| `rag/svr/task_executor_refactor/chunk_post_processor.py` | tag cache
restore |
| `rag/svr/task_executor_refactor/task_context.py` | language default
fix |
| `test/.../conftest.py` | +294 lines shared helpers |
| `test/.../*.py` | 15 test files refactored, 20 new tests |

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

rag/svr/task_executor_refactor/task_handler.py

@@ -39,6 +39,7 @@ from api.db.services.llm_service import LLMBundle
 from api.db.services.task_service import GRAPH_RAPTOR_FAKE_DOC_ID
 from common.constants import LLMType
 from common.exceptions import TaskCanceledException
+from common.connection_utils import timeout
 from common.misc_utils import thread_pool_exec
 from rag.nlp import search
 from rag.svr.task_executor_refactor.constants import CANVAS_DEBUG_DOC_ID
@@ -111,6 +112,7 @@ class TaskHandler:
                     logging.exception(
                         f"Remove doc({task_doc_id}) from docStore failed when task({task_id}) canceled, exception: {e}")
 
+    @timeout(60 * 60 * 3, 1)
     async def handle(self) -> None:
         """Handle a document processing task."""
         ctx = self._task_context
@@ -125,14 +127,6 @@ class TaskHandler:
             else:
                 # actual run - not dry run
                 await handle_save_to_memory_task(ctx.raw_task)
-
-        # Handle dataflow debug mode
-        if task_type == "dataflow" and ctx.doc_id == CANVAS_DEBUG_DOC_ID:
-            await self._run_dataflow()
-            return
-
-        if task_type.startswith("dataflow"):
-            await self._run_dataflow()
             return
 
         # Check if task is canceled
@@ -140,15 +134,25 @@ class TaskHandler:
             ctx.progress_cb(-1, msg="Task has been canceled.")
             return
 
-        # Bind embedding model
-        embedding_model = await self._bind_embedding_model()
-        if embedding_model is None:
+        # Language defaults to "Chinese" via TaskContext._DEFAULTS — safe to bind model directly.
+        # Bind embedding model (matching original do_handle_task order: bind + init_kb before routing)
+        result = await self._bind_embedding_model()
+        if result is None:
             return
+        embedding_model, vector_size = result
 
         with embedding_model:
-            vector_size = self._get_vector_size(embedding_model)
             self._init_kb(vector_size)
 
+            # Handle dataflow tasks (after init_kb, matching original behavior)
+            if task_type == "dataflow" and ctx.doc_id == CANVAS_DEBUG_DOC_ID:
+                await self._run_dataflow()
+                return
+
+            if task_type.startswith("dataflow"):
+                await self._run_dataflow()
+                return
+
             # Route to appropriate handler
             if task_type == "raptor":
                 await self._run_raptor(embedding_model, vector_size)
@@ -166,12 +170,6 @@ class TaskHandler:
                 await self._run_standard_chunking(embedding_model)
 
 
-    @classmethod
-    def _get_vector_size(cls, embedding_model: LLMBundle) -> int:
-        """Get vector size from embedding model."""
-        vts, _ = embedding_model.encode(["ok"])
-        return len(vts[0])
-
     def _init_kb(self, vector_size: int) -> None:
         """Initialize knowledge base index."""
         ctx = self._task_context
@@ -203,8 +201,12 @@ class TaskHandler:
         ctx = self._task_context
         ctx.progress_cb(1, "Clone task placeholder")
 
-    async def _bind_embedding_model(self) -> Optional[LLMBundle]:
-        """Bind embedding model to task."""
+    async def _bind_embedding_model(self) -> Optional[tuple]:
+        """Bind embedding model to task.
+
+        Returns:
+            Tuple of (embedding_model, vector_size) on success, or None on failure.
+        """
         ctx = self._task_context
         task_tenant_id = ctx.tenant_id
         task_embedding_id = ctx.embd_id
@@ -221,7 +223,7 @@ class TaskHandler:
                 )
             embedding_model = LLMBundle(task_tenant_id, embd_model_config, lang=task_language)
             vts, _ = embedding_model.encode(["ok"])
-            return embedding_model
+            return embedding_model, len(vts[0])
         except Exception as e:
             error_message = f'Fail to bind embedding model: {str(e)}'
             ctx.progress_cb(-1, msg=error_message)
@@ -340,8 +342,24 @@ class TaskHandler:
             kb_parser_config.update({
                 "graphrag": {
                     "use_graphrag": True,
-                    "entity_types": ["organization", "person", "geo", "event", "category"],
+                    "entity_types": [
+                        "organization",
+                        "person",
+                        "geo",
+                        "event",
+                        "category",
+                    ],
                     "method": "light",
+                    "batch_chunk_token_size": 4096,
+                    "retry_attempts": 2,
+                    "retry_backoff_seconds": 2.0,
+                    "retry_backoff_max_seconds": 60.0,
+                    "build_subgraph_timeout_per_chunk_seconds": 300,
+                    "build_subgraph_min_timeout_seconds": 600,
+                    "merge_timeout_seconds": 180,
+                    "resolution_timeout_seconds": 1800,
+                    "community_timeout_seconds": 1800,
+                    "lock_acquire_timeout_seconds": 600,
                 }
             })
             if ctx.write_interceptor:
@@ -400,6 +418,10 @@ class TaskHandler:
         # Get storage binary
         bucket, name = File2DocumentService.get_storage_address(doc_id=ctx.doc_id)
         binary = await self._get_storage_binary(bucket, name)
+        if binary is None:
+            raise FileNotFoundError(
+                f"Can not find file <{ctx.name}> from minio. Could you try it again."
+            )
 
         chunks = await chunk_service.build_chunks(binary)
         ctx.recording_context.record("chunks", chunks)
@@ -418,7 +440,7 @@ class TaskHandler:
         start_ts = timer()
         embedding_service = EmbeddingService(ctx=ctx)
         try:
-            token_count, vector_size = embedding_service.embed_chunks(
+            token_count, vector_size = await embedding_service.embed_chunks(
                 chunks, embedding_model, ctx.parser_config
             )
         except TaskCanceledException: