## Summary
RAGFlow's Go tokenizer silently returned **0 tokens for every string**
whenever the `cl100k_base` BPE table could not be loaded — which is the
normal case for an offline/air-gapped Go server. This PR makes the
loader resolve the table from disk (where RAGFlow actually ships it) and
fail loudly when it is genuinely missing.
## Root cause
`tiktoken-go`'s stock loader downloads the encoding table over HTTP and
caches it under `TIKTOKEN_CACHE_DIR`. That does not work for RAGFlow:
- `TIKTOKEN_CACHE_DIR` is exported **only inside the Python process**
(`common/token_utils.py`). `docker/entrypoint.sh` launches the Go binary
(`bin/ragflow_server`) from a shell, so the Go process never inherits
the variable.
- The Dockerfile *does* ship the table (under its sha1 name in the
working directory), but nothing told the Go side to look there.
- Reaching `openaipublic.blob.core.windows.net` at runtime is not an
option for air-gapped installs, and is unreliable where that host is
blocked.
The failure was **silent**: `NumTokensFromString` returns `0` when the
encoder fails to build, and a `sync.Once` memoizes that error for the
process lifetime. Every token count became `0`, so chunk merging never
crossed its token budget and an entire document collapsed into a single
chunk. Python has no such failure mode because its encoder is built at
import time (a missing table aborts startup instead of degrading).
## Fix
Register a local-only `BpeLoader` via `tiktoken.SetBpeLoader`
(`internal/tokenizer/bpe_loader.go`) that resolves the table from disk
**only**, in priority order:
1. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` (honored so operators who
already configured one keep working).
2. The working directory, the executable's directory, and all of their
ancestors — matching the Dockerfile layout (table under its sha1 name in
the install root).
3. A `ragflow_deps/<basename>` checkout produced by
`ragflow_deps/download_deps.py`.
It **never performs network I/O**. When nothing is found it returns an
error listing every path it tried (pointing at `download_deps.py` or
`TIKTOKEN_CACHE_DIR`), so a genuinely missing table fails loudly instead
of degrading to zero.
## Test plan
- `internal/tokenizer/bpe_loader_test.go` (unit tier, runs under `bash
build.sh --test ./internal/tokenizer/...`):
- Loader reads from `TIKTOKEN_CACHE_DIR`, `DATA_GYM_CACHE_DIR`, the
sha1-named file in the working dir, and the bundled `ragflow_deps/`
name.
- Explicit cache dir wins over the bundled vocab.
- A malformed table is reported as an error rather than skipped.
- A genuinely missing table reports the candidates it tried (no network
attempt).
- `NumTokensFromString` matches Python-derived anchors (`""`→0,
`"hello"`→1, `"hello world"`→2, `"hello, world!"`→4, `"世界"`→3, `"Hello
世界 🌍"`→8, `"RAGFlow"`→3).
## Notes
- `.github/workflows/tests.yml` currently excludes `internal/tokenizer`
from `go test`, so these tests do not run in CI. The tokenizer fix is
exercised in CI indirectly via the chunker package once a
token-count-sensitive parity case lands (tracked separately). Consider
including `internal/tokenizer` in CI as a follow-up.
- Supported deployments already ship the table (`download_deps.py` →
`ragflow_deps/cl100k_base.tiktoken`; Dockerfile → `<sha1>` in cwd), so
no `ENV` change is required for the fix to take effect. Setting `ENV
TIKTOKEN_CACHE_DIR` in the Dockerfile remains a cheap
belt-and-suspenders hardening that can be done separately.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Co-authored-by: CodeBuddy Code <noreply@cnb.cool>
Co-authored-by: CodeBuddy <noreply@tencent.com>
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
## What problem does this PR solve?
Go test files are never compiled in CI — only production binaries via
`go build`. This allowed a missing `"sort"` import in
`metadata_filter_test.go` to be merged without detection.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Changes
- Add `go test -count=1 ./internal/...` step after Go build in CI
workflow
- Fix missing `"sort"` import in `metadata_filter_test.go` (pre-existing
compile error)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Since database model is updated in python version, go server also need
to update
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
# RAGFlow Go Implementation Plan 🚀
This repository tracks the progress of porting RAGFlow to Go. We'll
implement core features and provide performance comparisons between
Python and Go versions.
## Implementation Checklist
- [x] User Management APIs
- [x] Dataset Management Operations
- [x] Retrieval Test
- [x] Chat Management Operations
- [x] Infinity Go SDK
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
