Commit Graph

8 Commits

Author SHA1 Message Date
euvre
3733da2773 fix: count distinct chunk IDs so doc.chunk_num matches actual indexed chunks (#17182) 2026-07-23 09:58:43 +08:00
qinling0210
0189ca3700 Port PR14140 and PR16881 to GO (#17102)
### Summary

Port
https://github.com/infiniflow/ragflow/pull/14140/
https://github.com/infiniflow/ragflow/pull/16881
2026-07-21 17:46:23 +08:00
Jack
c396ae912e Remove redundant PatchDSL and consolidate component-param injection into override_params (#17079)
## Summary

`Pipeline.Run` already accepts an `override_params` map (keyed by
`cpnID`, override-wins) that is merged into each component's params at
compile time via `canvas.WithOverrideParams`. `PatchDSL` performed a
byte-for-byte equivalent merge by baking `ParserConfig` into the DSL
text before compilation, so `runPipelineWithDSL` applied the same
`Doc.ParserConfig` twice with no effect.

## Change

- Delete the now-dead `PatchDSL` function
(`internal/ingestion/pipeline/pipeline.go`); it had a single call site.
- In `runPipelineWithDSL`, compile the DSL unchanged and pass the
extracted component params to `Run` as `override_params`.

## Behavioral note (important)

Previously `PatchDSL` was also the channel that delivered the tenant LLM
id into Extractor components: it baked `parserConfig` — after
`common.InjectExtractorLLMID(parserConfig, Tenant.LLMID)` had injected
`llm_id` — into the DSL. The old `Run` override_params re-cast
`Doc.ParserConfig` (which intentionally omits `llm_id`). So removing
`PatchDSL` while keeping that re-cast would have silently dropped LLM id
injection. This change passes the local `parserConfig` (the
LLMID-injected copy) to `Run`, preserving the prior behavior.

The DSL string returned by `runPipelineWithDSL` is only consumed by
`recordPipelineLog` (structure storage) and `ExtractPayload` (which
reads the terminal structure, not params), so returning the original DSL
is equivalent.

## Test plan

- `bash build.sh --test ./internal/ingestion/task/...
./internal/ingestion/pipeline/...` — both packages pass.
- `gofmt` clean; pre-commit hooks pass.

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-20 11:46:07 +08:00
Jack
10c00a9614 Feat: add built in DSL file API (#17003)
### Summary

1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
2026-07-18 11:00:08 +08:00
qinling0210
995e405e8c Support pipeline DSL modification through dataset configuration (backend) (#16991)
…end)

### Summary

Support pipeline DSL modification through dataset configuration
(backend)

Key modification: knowledgebase.parser_config

---------

Co-authored-by: yzc <yuzhichang@gmail.com>
2026-07-17 14:40:09 +08:00
Jack
e543ff02c4 Feat: nats message processing refactor and support built in DSL (#16873) 2026-07-15 14:53:16 +08:00
Jack
cfacaccad7 Refactor: message processing (#16852)
### Summary

1. refactor message processing
2. delete un-used componentIndexMap
3. unfold (delete) internal/ingestion/task/task_handler.go
2026-07-13 16:32:34 +08:00
Jack
3c0f8fc192 Refactor: refactor dataflow_service.go and guard nats message re-delivery (#16826)
### Summary

1. refactor dataflow_service.go 
2. guard nats message re-delivery
3. support document parse cancelling & re-run
2026-07-13 11:08:04 +08:00