4 Commits

Author	SHA1	Message	Date
euvre	d9a04ef702	fix: support auto mode in table parser document metadata aggregation (#15780) ... ### What problem does this PR solve? Table parser metadata aggregation previously only ran when `table_column_mode` was set to `manual`. In auto mode (default), all columns default to `"both"` role, meaning they should also be aggregated into document-level metadata for UI/chat filters. Additionally, the task snapshot could be stale — `table_column_names` are written to KB `parser_config` during `chunk()` but the task may have been created before that. Changes: - Renames `aggregate_table_manual_doc_metadata` → `aggregate_table_doc_metadata` - Supports both `"manual"` and `"auto"` `table_column_mode` (defaults to `"auto"`) - Reloads `table_column_names` from KB DB when missing from task snapshot - Removes the manual-only guard in `task_executor` and refactored `post_processor` - Updates all tests with new function name and adds auto mode test cases ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue)	2026-06-08 19:08:23 +08:00
Jack	b363146997	refactor: overhaul task executor with layered architecture and comprehensive test suite (#15471) ... ## Summary Decomposes the monolithic `task_executor.py` (1945 lines) into a 6-layer architecture with clear separation of concerns. The refactored code is functionally equivalent to the original, verified through 400 passing tests and a production-vs-dry-run comparison framework. ## Architecture ``` entry (task_manager) └─ orchestration (task_handler) ├─ services (chunk_service, embedding_service, dataflow_service, raptor_service, post_processor) │ └─ utilities (chunk_builder, chunk_post_processor, embedding_utils) └─ infrastructure (task_context, recording_context, interceptor) ``` Key design decisions: - **TaskContext** — typed facade over raw task dict, injects rate limiters + callbacks via composition - **RecordingContext + Comparator** — enables side-by-side production vs dry-run execution for safe migration - **NullRecordingContext** — zero-allocation no-op for production, uses `__slots__` - **WriteOperationInterceptor** — FIFO replay of previous runs function returns for comparison mode ## Migration Strategy The original `handle_task()` in `task_executor.py` uses a 3-way switch via `TE_RUN_MODE`: - `TE_RUN_MODE=0` (default) → runs refactored code - `TE_RUN_MODE=1` → runs both original + refactored, compares all intermediate results - `TE_RUN_MODE=2` → runs original code (fallback) The comparison mode (`TE_RUN_MODE=1`) records ~40 intermediate values (chunks, vectors, token counts, func return values) from the production run and replays them during dry-run, then uses `ContextComparator` to report mismatches. ## Functional Equivalence Fixes All divergences between original and refactored code were identified and fixed: - Timeout decorators (handle/build_chunks/raptor/embedding) - NullRecordingContext leak in finally block causing RuntimeError - MinIO None-binary check with proper FileNotFoundError - Dataflow dispatch after embedding binding + init_kb - Memory task missing return after processing - RAPTOR checkpoint progress reporting - Tag cache (get_tags_from_cache/set_tags_to_cache) restoration - dataflow_id correction in _load_dsl - Language default Chinese, dead code guard removal - embed_chunks made async with proper thread_pool_exec - Full GraphRAG default configuration (10 parameters) - Hardcoded q_768_vec fallback removal in RAPTOR ## Test Changes - 20 new tests covering table parser manual mode, tag cache, embedding edge cases, RAPTOR checkpoint, dataflow_id correction, storage binary None, cancel cleanup, metadata=None boundary - Unified `make_task_context`/`make_task_dict` factories eliminated 10+ duplicated helpers - DataflowService tests migrated from internal method mocks to IO boundary mocks (real orchestration code executes) - Parametrized duplicate build_chunks post-processor tests - 7 raptor tests modernized to @pytest.mark.asyncio - Mock count per test reduced through boundary-level mocking strategy **Test count: 400 passing, 0 warnings, 0 skips** ## Files Changed | File | Change | |------|--------| | `rag/svr/task_executor.py` | +1 line (NullRecordingContext fix) | | `rag/svr/task_executor_refactor/task_handler.py` | Orchestration layer, 8 logic fixes | | `rag/svr/task_executor_refactor/chunk_service.py` | +timeout + None-check | | `rag/svr/task_executor_refactor/embedding_service.py` | sync→async rewrite | | `rag/svr/task_executor_refactor/dataflow_service.py` | dataflow_id fix + timeout | | `rag/svr/task_executor_refactor/raptor_service.py` | checkpoint fix + assert | | `rag/svr/task_executor_refactor/chunk_post_processor.py` | tag cache restore | | `rag/svr/task_executor_refactor/task_context.py` | language default fix | | `test/.../conftest.py` | +294 lines shared helpers | | `test/.../*.py` | 15 test files refactored, 20 new tests | --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-03 17:18:31 +08:00
Lynn	dc4b82523b	Feat: tenant llm provider (#14595) ... ### What problem does this PR solve? Python implementation of the Go-based model_provider API suite. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: bill <yibie_jingnian@163.com>	2026-05-29 17:39:41 +08:00
Jack	f0cb7a544b	Refactor: Task Executor (#15154) ... ### What problem does this PR solve? 1. Break huge function into smaller pieces 2. Add unit test for the smaller pieces function 3. Layer-ed design a. infra layer - task_context.py, recording_context.py, write_operation_interceptor.py, ... b. service layer - *_service.py c. business layer - task_handler.py 4. Default behavior: use "refactor-ed version" - can switch to original version by change env variable ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [x] Refactoring - [x] Performance Improvement --------- Co-authored-by: Liu An <asiro@qq.com> Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>	2026-05-27 21:54:17 +08:00