zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-08-04 23:00:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(tokenizer): load cl100k BPE table from disk instead of failing silently offline (#17712)

Browse Source 
## Summary

RAGFlow's Go tokenizer silently returned **0 tokens for every string**
whenever the `cl100k_base` BPE table could not be loaded — which is the
normal case for an offline/air-gapped Go server. This PR makes the
loader resolve the table from disk (where RAGFlow actually ships it) and
fail loudly when it is genuinely missing.

## Root cause

`tiktoken-go`'s stock loader downloads the encoding table over HTTP and
caches it under `TIKTOKEN_CACHE_DIR`. That does not work for RAGFlow:

- `TIKTOKEN_CACHE_DIR` is exported **only inside the Python process**
(`common/token_utils.py`). `docker/entrypoint.sh` launches the Go binary
(`bin/ragflow_server`) from a shell, so the Go process never inherits
the variable.
- The Dockerfile *does* ship the table (under its sha1 name in the
working directory), but nothing told the Go side to look there.
- Reaching `openaipublic.blob.core.windows.net` at runtime is not an
option for air-gapped installs, and is unreliable where that host is
blocked.

The failure was **silent**: `NumTokensFromString` returns `0` when the
encoder fails to build, and a `sync.Once` memoizes that error for the
process lifetime. Every token count became `0`, so chunk merging never
crossed its token budget and an entire document collapsed into a single
chunk. Python has no such failure mode because its encoder is built at
import time (a missing table aborts startup instead of degrading).

## Fix

Register a local-only `BpeLoader` via `tiktoken.SetBpeLoader`
(`internal/tokenizer/bpe_loader.go`) that resolves the table from disk
**only**, in priority order:

1. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` (honored so operators who
already configured one keep working).
2. The working directory, the executable's directory, and all of their
ancestors — matching the Dockerfile layout (table under its sha1 name in
the install root).
3. A `ragflow_deps/<basename>` checkout produced by
`ragflow_deps/download_deps.py`.

It **never performs network I/O**. When nothing is found it returns an
error listing every path it tried (pointing at `download_deps.py` or
`TIKTOKEN_CACHE_DIR`), so a genuinely missing table fails loudly instead
of degrading to zero.

## Test plan

- `internal/tokenizer/bpe_loader_test.go` (unit tier, runs under `bash
build.sh --test ./internal/tokenizer/...`):
- Loader reads from `TIKTOKEN_CACHE_DIR`, `DATA_GYM_CACHE_DIR`, the
sha1-named file in the working dir, and the bundled `ragflow_deps/`
name.
  - Explicit cache dir wins over the bundled vocab.
  - A malformed table is reported as an error rather than skipped.
- A genuinely missing table reports the candidates it tried (no network
attempt).
- `NumTokensFromString` matches Python-derived anchors (`""`→0,
`"hello"`→1, `"hello world"`→2, `"hello, world!"`→4, `"世界"`→3, `"Hello
世界 🌍"`→8, `"RAGFlow"`→3).

## Notes

- `.github/workflows/tests.yml` currently excludes `internal/tokenizer`
from `go test`, so these tests do not run in CI. The tokenizer fix is
exercised in CI indirectly via the chunker package once a
token-count-sensitive parity case lands (tracked separately). Consider
including `internal/tokenizer` in CI as a follow-up.
- Supported deployments already ship the table (`download_deps.py` →
`ragflow_deps/cl100k_base.tiktoken`; Dockerfile → `<sha1>` in cwd), so
no `ENV` change is required for the fix to take effect. Setting `ENV
TIKTOKEN_CACHE_DIR` in the Dockerfile remains a cheap
belt-and-suspenders hardening that can be done separately.

🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)

---------

Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Co-authored-by: CodeBuddy Code <noreply@cnb.cool>
Co-authored-by: CodeBuddy <noreply@tencent.com>
This commit is contained in:
Jack
2026-08-03 19:03:08 +08:00
committed by GitHub
parent a75e733b39
commit e997fd655a
7 changed files with 673 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
.github/workflows/sep-tests.yml vendored
View File

@@ -320,14 +320,55 @@ jobs:
#
# Excludes packages whose tests fail for environmental reasons
# unrelated to the diff:
# - internal/tokenizer: tests need /usr/share/infinity/resource
# dict files, only mounted inside the docker builder, not
# in the Go test environment.
# - internal/tokenizer is split: the pure-Go BPE loader tests
# (bpe_loader_test.go) run in the default tier; the C++ binding /
# dict-dependent tests (tokenizer_test.go,
# tokenizer_concurrent_test.go) and the on-disk anchor test
# (bpe_loader_anchors_test.go) are tagged `manual` and need the
# docker builder's /usr/share/infinity/resource, so they stay out
# of the default run.
run: |
set -euo pipefail
# Provide the cl100k BPE table for the offline tokenizer loader.
# The loader reads <repo>/ragflow_deps/cl100k_base.tiktoken (or
# TIKTOKEN_CACHE_DIR). Try to fetch it directly from the upstream
# openai blob first (small ~1.6MB, reachable from the CI runner — the
# pre-PR code downloaded it on the fly, so the network path is known
# good). If the direct fetch fails (e.g. GFW), fall back to extracting
# it from the local infiniflow/ragflow_deps image *if already present*
# on the runner (no `docker pull`, since that huge image is first
# fetched at the later "Build ragflow:nightly" step). Best-effort: if
# neither source works, the loader still fails loudly (no silent
# 0-token degradation).
if [ ! -f ragflow_deps/cl100k_base.tiktoken ]; then
mkdir -p ragflow_deps
if command -v curl >/dev/null 2>&1; then
if curl -fsSL -o ragflow_deps/cl100k_base.tiktoken \
https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken; then
echo "tiktoken: cl100k table fetched via curl from openai blob"
else
echo "tiktoken: curl fetch failed (network/GFW); will try cached image"
fi
fi
if [ ! -f ragflow_deps/cl100k_base.tiktoken ] && docker image inspect infiniflow/ragflow_deps:latest >/dev/null 2>&1; then
CID=$(docker create infiniflow/ragflow_deps:latest true) || true
if [ -n "${CID:-}" ]; then
if docker cp "$CID":/cl100k_base.tiktoken ragflow_deps/cl100k_base.tiktoken 2>/dev/null; then
echo "tiktoken: cl100k table copied from cached infiniflow/ragflow_deps image"
fi
docker rm -f "$CID" >/dev/null 2>&1 || true
fi
fi
fi
if [ -f ragflow_deps/cl100k_base.tiktoken ]; then
echo "tiktoken: cl100k table provisioned ($(wc -c < ragflow_deps/cl100k_base.tiktoken) bytes)"
else
echo "tiktoken: cl100k table NOT provisioned — offline loader will fail loudly"
fi
PKGS=$(go list ./... 2>/dev/null \
| grep -v '/internal/storage$' \
| grep -v '/internal/tokenizer$' \
| grep -v '/internal/handler$' || true)
if [ -z "$PKGS" ]; then
./build.sh --test
@@ -895,14 +936,55 @@ jobs:
#
# Excludes packages whose tests fail for environmental reasons
# unrelated to the diff:
# - internal/tokenizer: tests need /usr/share/infinity/resource
# dict files, only mounted inside the docker builder, not
# in the Go test environment.
# - internal/tokenizer is split: the pure-Go BPE loader tests
# (bpe_loader_test.go) run in the default tier; the C++ binding /
# dict-dependent tests (tokenizer_test.go,
# tokenizer_concurrent_test.go) and the on-disk anchor test
# (bpe_loader_anchors_test.go) are tagged `manual` and need the
# docker builder's /usr/share/infinity/resource, so they stay out
# of the default run.
run: |
set -euo pipefail
# Provide the cl100k BPE table for the offline tokenizer loader.
# The loader reads <repo>/ragflow_deps/cl100k_base.tiktoken (or
# TIKTOKEN_CACHE_DIR). Try to fetch it directly from the upstream
# openai blob first (small ~1.6MB, reachable from the CI runner — the
# pre-PR code downloaded it on the fly, so the network path is known
# good). If the direct fetch fails (e.g. GFW), fall back to extracting
# it from the local infiniflow/ragflow_deps image *if already present*
# on the runner (no `docker pull`, since that huge image is first
# fetched at the later "Build ragflow:nightly" step). Best-effort: if
# neither source works, the loader still fails loudly (no silent
# 0-token degradation).
if [ ! -f ragflow_deps/cl100k_base.tiktoken ]; then
mkdir -p ragflow_deps
if command -v curl >/dev/null 2>&1; then
if curl -fsSL -o ragflow_deps/cl100k_base.tiktoken \
https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken; then
echo "tiktoken: cl100k table fetched via curl from openai blob"
else
echo "tiktoken: curl fetch failed (network/GFW); will try cached image"
fi
fi
if [ ! -f ragflow_deps/cl100k_base.tiktoken ] && docker image inspect infiniflow/ragflow_deps:latest >/dev/null 2>&1; then
CID=$(docker create infiniflow/ragflow_deps:latest true) || true
if [ -n "${CID:-}" ]; then
if docker cp "$CID":/cl100k_base.tiktoken ragflow_deps/cl100k_base.tiktoken 2>/dev/null; then
echo "tiktoken: cl100k table copied from cached infiniflow/ragflow_deps image"
fi
docker rm -f "$CID" >/dev/null 2>&1 || true
fi
fi
fi
if [ -f ragflow_deps/cl100k_base.tiktoken ]; then
echo "tiktoken: cl100k table provisioned ($(wc -c < ragflow_deps/cl100k_base.tiktoken) bytes)"
else
echo "tiktoken: cl100k table NOT provisioned — offline loader will fail loudly"
fi
PKGS=$(go list ./... 2>/dev/null \
| grep -v '/internal/storage$' \
| grep -v '/internal/tokenizer$' \
| grep -v '/internal/handler$' || true)
if [ -z "$PKGS" ]; then
./build.sh --test

96
.github/workflows/tests.yml vendored
View File

@@ -197,15 +197,54 @@ jobs:
#
# Excludes packages whose tests fail for environmental reasons
# unrelated to the diff:
# - internal/tokenizer: tests need /usr/share/infinity/resource
# dict files, only mounted inside the docker builder, not
# in the Go test environment.
# - internal/tokenizer is split: the pure-Go BPE loader tests
# (bpe_loader_test.go) run in the default tier; the C++ binding /
# dict-dependent tests (tokenizer_test.go,
# tokenizer_concurrent_test.go) and the on-disk anchor test
# (bpe_loader_anchors_test.go) are tagged `manual` and need the
# docker builder's /usr/share/infinity/resource, so they stay out
# of the default run.
run: |
set -euo pipefail
# Provide the cl100k BPE table for the offline tokenizer loader.
# The loader reads <repo>/ragflow_deps/cl100k_base.tiktoken (or
# TIKTOKEN_CACHE_DIR). Fetch it directly: this CI runner can reach
# the upstream blob (pre-PR tests used tiktoken-go's stock network
# loader and passed), so a direct download is small and fast. As a
# fallback, copy from the infiniflow/ragflow_deps image only if it is
# already cached locally (we do NOT pull the whole image just for one
# 1.6MB file). If neither works, the loader still fails loudly (no
# silent 0-token degradation). The loader itself stays offline.
if [ ! -f ragflow_deps/cl100k_base.tiktoken ]; then
mkdir -p ragflow_deps
if command -v curl >/dev/null 2>&1; then
if curl -fsSL -o ragflow_deps/cl100k_base.tiktoken \
https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken; then
echo "tiktoken: cl100k table fetched via curl from openai blob"
else
echo "tiktoken: curl fetch failed (network/GFW); will try cached image"
fi
fi
if [ ! -f ragflow_deps/cl100k_base.tiktoken ] && docker image inspect infiniflow/ragflow_deps:latest >/dev/null 2>&1; then
CID=$(docker create infiniflow/ragflow_deps:latest true) || true
if [ -n "${CID:-}" ]; then
if docker cp "$CID":/cl100k_base.tiktoken ragflow_deps/cl100k_base.tiktoken 2>/dev/null; then
echo "tiktoken: cl100k table copied from cached infiniflow/ragflow_deps image"
fi
docker rm -f "$CID" >/dev/null 2>&1 || true
fi
fi
fi
if [ -f ragflow_deps/cl100k_base.tiktoken ]; then
echo "tiktoken: cl100k table provisioned ($(wc -c < ragflow_deps/cl100k_base.tiktoken) bytes)"
else
echo "tiktoken: cl100k table NOT provisioned — offline loader will fail loudly"
fi
PKGS=$(go list ./... 2>/dev/null \
| grep -v '/internal/storage$' \
| grep -v '/internal/agent$' \
| grep -v '/internal/tokenizer$' \
| grep -v '/internal/handler$' || true)
if [ -z "$PKGS" ]; then
./build.sh --test
@@ -640,13 +679,52 @@ jobs:
#
# Excludes packages whose tests fail for environmental reasons
# unrelated to the diff:
# - internal/tokenizer: tests need /usr/share/infinity/resource
# dict files, only mounted inside the docker builder, not
# in the Go test environment.
# - internal/tokenizer is split: the pure-Go BPE loader tests
# (bpe_loader_test.go) run in the default tier; the C++ binding /
# dict-dependent tests (tokenizer_test.go,
# tokenizer_concurrent_test.go) and the on-disk anchor test
# (bpe_loader_anchors_test.go) are tagged `manual` and need the
# docker builder's /usr/share/infinity/resource, so they stay out
# of the default run.
run: |
set -euo pipefail
PKGS=$(go list ./... 2>/dev/null \
| grep -v '/internal/tokenizer$' || true)
# Provide the cl100k BPE table for the offline tokenizer loader.
# The loader reads <repo>/ragflow_deps/cl100k_base.tiktoken (or
# TIKTOKEN_CACHE_DIR). Fetch it directly: this CI runner can reach
# the upstream blob (pre-PR tests used tiktoken-go's stock network
# loader and passed), so a direct download is small and fast. As a
# fallback, copy from the infiniflow/ragflow_deps image only if it is
# already cached locally (we do NOT pull the whole image just for one
# 1.6MB file). If neither works, the loader still fails loudly (no
# silent 0-token degradation). The loader itself stays offline.
if [ ! -f ragflow_deps/cl100k_base.tiktoken ]; then
mkdir -p ragflow_deps
if command -v curl >/dev/null 2>&1; then
if curl -fsSL -o ragflow_deps/cl100k_base.tiktoken \
https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken; then
echo "tiktoken: cl100k table fetched via curl from openai blob"
else
echo "tiktoken: curl fetch failed (network/GFW); will try cached image"
fi
fi
if [ ! -f ragflow_deps/cl100k_base.tiktoken ] && docker image inspect infiniflow/ragflow_deps:latest >/dev/null 2>&1; then
CID=$(docker create infiniflow/ragflow_deps:latest true) || true
if [ -n "${CID:-}" ]; then
if docker cp "$CID":/cl100k_base.tiktoken ragflow_deps/cl100k_base.tiktoken 2>/dev/null; then
echo "tiktoken: cl100k table copied from cached infiniflow/ragflow_deps image"
fi
docker rm -f "$CID" >/dev/null 2>&1 || true
fi
fi
fi
if [ -f ragflow_deps/cl100k_base.tiktoken ]; then
echo "tiktoken: cl100k table provisioned ($(wc -c < ragflow_deps/cl100k_base.tiktoken) bytes)"
else
echo "tiktoken: cl100k table NOT provisioned — offline loader will fail loudly"
fi
PKGS=$(go list ./... 2>/dev/null || true)
if [ -z "$PKGS" ]; then
./build.sh --test
else