mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Sanderhoff-alt
a6a6c8e5a1
* docs(docker): add guide and recipe for building CUDA standalone image Provide a Docker Compose example and standalone Dockerfile recipe for building a CUDA-enabled PyTorch image for NVIDIA GPU-accelerated local embedding and reranker models. Document prerequisites, build steps, and NVIDIA Container Toolkit configuration. * docs(docker): correct CUDA recipe verification, drop x86-only pin, state size Review follow-ups on the CUDA recipe: - The README told users to verify GPU placement by grepping the logs for both the embedding and cross-encoder device, but only the embedder logged one. LocalSTCrossEncoder resolved _device_type and never reported it, so the documented check showed a single line and looked like the reranker was still on CPU. Log the device on both reranker init paths and show the real log output in the README. - Dropped the hardcoded `--platform=linux/amd64`. PyTorch ships cu126 wheels for aarch64 too, and on an arm64 host the pin silently produced an emulated amd64 image that cannot reach the GPU at all. Documented instead that the build must be native. - Dropped `--index-strategy unsafe-best-match`. It relaxed the index isolation that hindsight-api-slim/pyproject.toml deliberately sets up, and it was not needed: resolving without it succeeds and yields the same package set. - Documented the image size (~11 GB vs ~9 GB for the base) in installation.md and the recipe README, since that cost is the reason no CUDA image is published. - Added a HINDSIGHT_VERSION build arg so the base tag can be pinned, fixed the manual `docker build` context, and enabled RERANKER_LOCAL_FP16 in the compose file (faster on GPU, quality-identical). Verified: `uv pip install` inside the base image replaces only torch (2.10.0+cpu -> 2.10.0+cu126) and adds the nvidia/cuda runtime wheels; compose config validates; docs skill regenerates clean; test_local_cross_encoder.py passes (21). Claude-Session: https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
54 lines
1.7 KiB
YAML
54 lines
1.7 KiB
YAML
name: hindsight-cuda
|
|
# Example: run Hindsight with CUDA GPU acceleration for in-process
|
|
# local embedding and reranker models.
|
|
#
|
|
# Requirements on the host:
|
|
# - NVIDIA GPU & compatible driver (>= 525.60.13)
|
|
# - NVIDIA Container Toolkit installed & configured
|
|
#
|
|
# Quick start:
|
|
# export HINDSIGHT_API_LLM_API_KEY=sk-xxx
|
|
# docker compose -f docker/docker-compose/cuda/docker-compose.yaml up --build
|
|
#
|
|
# Required environment variables:
|
|
# - HINDSIGHT_API_LLM_API_KEY (pair it with HINDSIGHT_API_LLM_PROVIDER to use
|
|
# a provider other than the default openai)
|
|
|
|
services:
|
|
hindsight:
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
args:
|
|
# Pin to a released tag (e.g. 0.9.2) instead of latest for reproducible builds.
|
|
HINDSIGHT_VERSION: ${HINDSIGHT_VERSION:-latest}
|
|
PYTORCH_CUDA_FLAVOR: cu126
|
|
PYTORCH_CUDA_INDEX: https://download.pytorch.org/whl/cu126
|
|
container_name: hindsight-cuda
|
|
ports:
|
|
- "8888:8888"
|
|
- "9999:9999"
|
|
environment:
|
|
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
|
|
HINDSIGHT_API_LLM_API_KEY: ${HINDSIGHT_API_LLM_API_KEY:-}
|
|
|
|
# Local embeddings & reranker will automatically detect and use CUDA.
|
|
HINDSIGHT_API_EMBEDDINGS_PROVIDER: local
|
|
HINDSIGHT_API_RERANKER_PROVIDER: local
|
|
|
|
# Half-precision reranking: measurably faster on GPU, quality-identical.
|
|
# Off by default because some CPUs lack native FP16 support.
|
|
HINDSIGHT_API_RERANKER_LOCAL_FP16: "true"
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
count: all
|
|
capabilities: [gpu]
|
|
volumes:
|
|
- pg_data:/home/hindsight/.pg0
|
|
|
|
volumes:
|
|
pg_data:
|