Sanderhoff-alt
a6a6c8e5a1
* docs(docker): add guide and recipe for building CUDA standalone image Provide a Docker Compose example and standalone Dockerfile recipe for building a CUDA-enabled PyTorch image for NVIDIA GPU-accelerated local embedding and reranker models. Document prerequisites, build steps, and NVIDIA Container Toolkit configuration. * docs(docker): correct CUDA recipe verification, drop x86-only pin, state size Review follow-ups on the CUDA recipe: - The README told users to verify GPU placement by grepping the logs for both the embedding and cross-encoder device, but only the embedder logged one. LocalSTCrossEncoder resolved _device_type and never reported it, so the documented check showed a single line and looked like the reranker was still on CPU. Log the device on both reranker init paths and show the real log output in the README. - Dropped the hardcoded `--platform=linux/amd64`. PyTorch ships cu126 wheels for aarch64 too, and on an arm64 host the pin silently produced an emulated amd64 image that cannot reach the GPU at all. Documented instead that the build must be native. - Dropped `--index-strategy unsafe-best-match`. It relaxed the index isolation that hindsight-api-slim/pyproject.toml deliberately sets up, and it was not needed: resolving without it succeeds and yields the same package set. - Documented the image size (~11 GB vs ~9 GB for the base) in installation.md and the recipe README, since that cost is the reason no CUDA image is published. - Added a HINDSIGHT_VERSION build arg so the base tag can be pinned, fixed the manual `docker build` context, and enabled RERANKER_LOCAL_FP16 in the compose file (faster on GPU, quality-identical). Verified: `uv pip install` inside the base image replaces only torch (2.10.0+cpu -> 2.10.0+cu126) and adds the nvidia/cuda runtime wheels; compose config validates; docs skill regenerates clean; test_local_cross_encoder.py passes (21). Claude-Session: https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
Hindsight with NVIDIA CUDA GPU Acceleration
Example setup that builds a custom Hindsight image with CUDA-enabled PyTorch for NVIDIA GPU-accelerated local embeddings and reranking.
When to use this
- You want to use in-process local embeddings (
HINDSIGHT_API_EMBEDDINGS_PROVIDER: local) and reranking (HINDSIGHT_API_RERANKER_PROVIDER: local) with NVIDIA GPU acceleration. - You want lower latency and higher throughput for local embedding and reranker inference.
- You have an NVIDIA GPU and want to run Hindsight locally without external TEI sidecars.
Note
This accelerates Hindsight's in-process PyTorch embedding and reranker models. The LLM (used for retain/recall/reflect) is external by default (e.g. OpenAI, Anthropic, Ollama, vLLM).
The CUDA runtime is added on top of the full image, whose CPU PyTorch wheel stays in the base layers. Expect the result to be roughly 11 GB on disk, against ~9 GB for the base image it builds on.
Prerequisites
- NVIDIA GPU with compatible driver (driver version
>= 525.60.13recommended for CUDA 12.x). - NVIDIA Container Toolkit installed and configured on the host Docker daemon.
- PyTorch ships CUDA wheels for both
x86_64andaarch64, so either architecture works. Build on the machine that will run the image — an emulated cross-architecture build cannot reach the GPU.
Verify GPU access in Docker:
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
Quick start
export HINDSIGHT_API_LLM_API_KEY=sk-xxx
docker compose -f docker/docker-compose/cuda/docker-compose.yaml up --build
- API: http://localhost:8888
- Control Plane: http://localhost:9999
To build against a pinned release rather than latest, set HINDSIGHT_VERSION:
HINDSIGHT_VERSION=0.9.2 docker compose -f docker/docker-compose/cuda/docker-compose.yaml up --build
Building manually
You can also build the image directly using docker build:
docker build -t hindsight:cuda docker/docker-compose/cuda/
Then run the container with GPU passthrough:
docker run --gpus all \
--name hindsight-cuda \
-p 8888:8888 -p 9999:9999 \
-e HINDSIGHT_API_LLM_API_KEY=sk-xxx \
hindsight:cuda
Verifying CUDA GPU Acceleration
The build itself fails if the CUDA wheel did not land, so a successful build already proves PyTorch has a CUDA runtime. To confirm the models actually loaded onto the GPU, check the container logs:
docker logs hindsight-cuda | grep -i "device:"
Both the embedding and the reranker provider report their device on startup, and
both should read device: cuda rather than device: cpu:
Embeddings: local provider initialized (dim: 384, device: cuda)
Reranker: local provider initialized (device: cuda, max_concurrent=4)
You can also query PyTorch directly inside the running container:
docker exec hindsight-cuda python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
Tuning
HINDSIGHT_API_RERANKER_LOCAL_FP16— half-precision reranking, enabled in the compose file above. Measurably faster on GPU and quality-identical; it is off by default only because some CPUs lack native FP16 support.HINDSIGHT_API_RERANKER_LOCAL_BATCH_SIZE— optimal batch size varies by GPU and model; worth tuning if reranking dominates your recall latency.
See Configuration for the full set of knobs.