## Summary
- align the Go Google Scholar component with the Python-side config
pattern
- merge node-level params with runtime inputs so canvas defaults are
preserved and per-run inputs can override them
- add tests covering node param fallback and runtime override behavior
## Verification
- `bash build.sh --test ./internal/agent/component/... -run
TestGoogleScholar`
<img width="1873" height="1165" alt="image"
src="https://github.com/user-attachments/assets/67198c6f-6a0e-43bf-a500-8e88d82b8751"
/>
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
## Summary
- register the Go `DuckDuckGo` canvas component and restore its dynamic
input form metadata
- align the Go component input/output surface with the current canvas
usage for `query`, `channel`, and `top_n`
- fix DuckDuckGo news search in Go by fetching the required `vqd` token
before calling `news.js`, and add targeted regression tests
## Testing
Passed:
- `bash build.sh --test ./internal/agent/tool/... -run 'DuckDuckGo'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo|TestVerifyRegistration_P1'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo'`
Not run:
- frontend tests
- frontend build
- full Go test suite
<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/9f3f8e4b-f6b4-4915-b96c-3c5b8c7b8b30"
/>
### Summary
- Implemented googleComponent wrapper to bridge the canvas component
contract with Eino's SerpApi-backed GoogleTool.
- Added parameter alias mapping (query to q, max_results to num) and
content formatting logic to match Python search result representation.
- Registered the "Google" component and the "google" tool factory in the
Go agent runtime to support web search nodes.
<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/e295ab88-e48c-4fe2-bcb7-47ca5b977c9b"
/>
### Summary
- TavilySearch now stores api_key from component params and injects it
into tool calls when runtime inputs omit it.
- TavilyExtract and BGPT now follow the same stored api_key behavior.
- Canvas decoding now recovers api_key from graph.nodes[].data.form when
components[].obj.params.api_key is empty, matching frontend payload
behavior without changing frontend data.
- Added regression tests for graph form key recovery and stored key
injection / caller key precedence.
Tests: build.sh --test ./internal/agent/component ./internal/service —
all pass.
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/0be31587-c1ba-4f3e-b43a-4fe0fca5a44c"
/>
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/e3edd92c-c62e-4db4-abe2-772bdf4fe1b2"
/>
### Summary
Handle searching dataset without embedding model
In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
## Summary
This PR updates Go agent publish logic to persist the parent canvas
update and canvas-version save in the same transaction.
## Changes:
- Reuse SaveOrReplaceLatest semantics for published versions
- Add SaveOrReplaceLatestTx for transactional publish flow
- Keep canvas release update and version persistence atomic
- Add a focused publish test covering canvas and released version state
## Tested:
```
bash build.sh --test -run 'Test(PublishAgentUpdatesCanvasAndReleasedVersion|UpdateAgentDSLCreatesAndReplacesDraftVersion|
UpdateAgentDSLDoesNotOverwriteLatestReleasedVersion)$' ./internal/service ./internal/dao
```
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/2c576581-1143-420b-8750-a77aa3c4292d"
/>
### Summary
The Go backend never seeded the `canvas_template` table, so the "Create
agent from template" page was blank when the frontend proxies to the Go
API (`API_PROXY_SCHEME=go`). This PR adds `SeedCanvasTemplates()` in
`internal/dao`, invoked from `InitDB()` after migrations, which loads
`agent/templates/*.json` and mirrors Python's `add_graph_templates()`
behavior.
Changes:
- Add `internal/dao/canvas_template_seed.go` to parse and upsert
built-in templates.
- Call `SeedCanvasTemplates()` in `InitDB()`.
- Add `CanvasTypes` (`JSONSlice`) to `entity.CanvasTemplate` so the
frontend can filter/group by category.
- Skip seeding gracefully when the templates directory is absent.
This fixes the blank template catalogue in Go mode.
## Summary
Adds the missing input form metadata for the Go BGPT canvas component.
## Root Cause
The standalone BGPT component was registered in Go, but it did not
implement GetInputForm(). During component trial run, the backend asks
the component for its input_form. Since BGPT had none, the API returned:
component has no input_form: BGPT:<node_id>
Python BGPT already exposes the query input form, so the Go component
needed the same contract.
## Change
Added GetInputForm() to the Go BGPT component with a single query line
input.
Added test coverage to ensure BGPT exposes the input form.
## Validation
Backend:
bash build.sh --test -run TestBGPT ./internal/agent/component
<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/f99e4a81-2359-42e5-80bb-dcc4e6a63fea"
/>
<img width="1736" height="1152" alt="image"
src="https://github.com/user-attachments/assets/c11240a5-2c42-4d08-88e3-c6dfbf49eedb"
/>
### Motivation
This PR evolves the harness from a pure execution runtime into an
**observable, replayable agent evaluation platform**. The current
`harness/graph` checkpoint mechanism is insufficient for true
event-sourced introspection—we need append-only event logs capturing
every tool call, state transition, memory write, and approval decision,
enabling deterministic replay, fork/diff, postmortem analysis, and
time-travel debugging.
### Key Design Goals
1. **Event-Sourced Execution Model**
Replace coarse checkpoints with granular, append-only event logs. Every
operation becomes a durable event: tool invocation, state mutation,
memory update, human approval. This unlocks deterministic replay,
branching execution histories, and regression datasets derived directly
from production failures.
2. **First-Class Replay & Evaluation Loop**
Replay is not an afterthought—it is a core primitive. A single live run
seeds an offline corpus that supports: repeated playback, model
substitution, tool result mocking, and strategy comparison. The harness
graduates from "executor" to "continuous evaluation platform" where
failed production traces convert directly into offline regression
suites.
3. **Operational Observability**
Beyond raw traces, expose metrics that prove stability over time:
- Tool success / failure rates
- Approval latency distributions
- Retry frequencies
- Checkpoint restore reliability
- Memory retrieval quality
- Cost per completed task
- Fork replay pass rates
The underlying thesis: the bottleneck for most agent systems is not
execution capability, but the inability to **demonstrate continuous,
measurable improvement**.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Ports remaining Go parser wiring and PDF backends, adds tenant-aware VLM
dispatch, aligns post-processing with Python, and adds end-to-end
pipeline coverage with a generated six-page PDF.
### Summary
Keep `data` as the uploaded document array when dataset document upload
partially succeeds.
This matches the Python API behavior and allows parse-on-creation to run
for successfully uploaded files when other files in the same folder are
unsupported.
## Summary
- Add Go dynamic input form support for ExeSQL and Browser components.
- Align their input form metadata with the Python implementation.
- Add regression tests for `/components/:component_id/input-form`.
## Summary
Debugging YahooFinance component in agent canvas returns "unknown
component" and "no input_form".
YahooFinance was only registered as an eino tool, not as a runtime
component. The component factory only searches the runtime registry.
- `universe_a_wrappers.go`: add `yahooFinanceComponent` wrapper
delegating to `agenttool.YahooFinanceTool` with `GetInputForm()`
- `fixture_stubs.go`: register `"YahooFinance"` component
## TEST
`go build` and `go test ./internal/agent/component/...` all pass.
### What problem does this PR solve?
The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).
This adds two optional, default-off booleans to the `filter` config:
- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).
They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16048
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ling Qin <qinling0210@163.com>
### What problem does this PR solve?
The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.
This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:
- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.
The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.
It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16046
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>
## Summary
- derive Go Agent debug input forms from prompt variable references
instead of Agent meta fields
- seed `sys.*` debug params into `CanvasState.Sys` so single-component
debug resolves prompt variables like Python
- restore Agent test-run parity for form rendering and debug execution
## Tests
- `go test ./internal/agent/component -run
'TestAgent_(GetInputForm_UsesPromptReferences|GetInputForm_DeduplicatesPromptReferences|Meta_DefaultsToEmpty|Reset_NoTools)$'`
- `go test ./internal/handler -run
'Test(DebugComponent_SeedsSysInputsIntoCanvasState|DebugComponent_HappyPath_Begin|GetComponentInputForm_HappyPath)$'`
AFTER:
<img width="669" height="456" alt="image"
src="https://github.com/user-attachments/assets/4fd86559-aafc-4027-91ae-6e666137ee1b"
/>
## Summary
Adds ownership/access checks before updating or deleting documents,
setting document metadata, and reading file contents from storage. Also
adds tests for authorized and unauthorized access paths.
## Summary
Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.
`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.
## Testing
- Ran Go unit tests:
```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```
- Frontend verification:
1. Open /files/skills.
2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
4. Search by skill name or description keyword.
5. Confirm matching skills are returned.
### Summary
This PR fixes two issues that prevented the Agent component's
single-component debug/test run from working under the Go backend:
1. **Dynamic input_form generation**: Some components (e.g. `Agent`) do
not store a static `input_form` in the DSL. The Go handler now falls
back to the runtime component's `GetInputForm()` method, matching
Python's `Canvas.get_component_input_form` behavior. This resolves the
frontend 102 error: `component has no input_form`.
2. **Tenant ID injection for debug**: Single-component debug runs use a
fresh `CanvasState` that previously lacked `tenant_id`.
`AgentComponent.Invoke` resolves LLM credentials via the tenant tables,
so the debug run failed with `api key is required`. The handler now
seeds `state.Sys["tenant_id"]` with the authenticated user's ID,
mirroring Python's `@add_tenant_id_to_kwargs` decorator.
### Changes
- `internal/handler/agent_component.go`:
- Added `componentInputForm` helper that first reads the static
`input_form` and, if missing, instantiates the component and calls
`GetInputForm()`.
- In `DebugComponent`, set `debugState.Sys["tenant_id"] = user.ID`
before invoking the component.
### Summary
The Go backend Agent component was not returning artifacts produced by
the CodeExec tool. While the Python agent collects the "`_ARTIFACTS`"
envelope from tool responses and appends artifact markdown to the final
content, the Go agent only returned the assistant text, so generated
images were missing from the chat output.
### Changes
- Wire `react.WithMessageFuture()` in `runEinoReActAgent` and store the
resulting `MessageFuture` in the invocation context.
- After the ReAct loop finishes, drain the future and extract
``_ARTIFACTS`` entries from every tool response message.
- Support reading the tool payload from both `msg.Content` and
`msg.UserInputMultiContent` to match eino's tool contract.
- De-duplicate artifacts by URL and render images as `!` and other files
as download links.
- Add `agent_artifact_test.go` with a regression test that simulates a
CodeExec-style tool response carrying an image artifact and verifies it
is collected and formatted.
### Verification
- `go test ./internal/agent/component/... -run
TestAgent_ReActAgent_CollectsArtifactsFromCodeExecTool` passes.
- `go test ./internal/agent/component/... -count=1` compiles; the only
failure is an unrelated DNS-pinning timeout test
(`TestInvoke_ProxyDNSPin`).
- `gofmt` clean for modified files.
### Related
Fixes the behavior shown in the screenshot where the Go agent ignored
the CodeExec-generated PNG artifact.
