mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Sanderhoff-alt
a6a6c8e5a1
* docs(docker): add guide and recipe for building CUDA standalone image Provide a Docker Compose example and standalone Dockerfile recipe for building a CUDA-enabled PyTorch image for NVIDIA GPU-accelerated local embedding and reranker models. Document prerequisites, build steps, and NVIDIA Container Toolkit configuration. * docs(docker): correct CUDA recipe verification, drop x86-only pin, state size Review follow-ups on the CUDA recipe: - The README told users to verify GPU placement by grepping the logs for both the embedding and cross-encoder device, but only the embedder logged one. LocalSTCrossEncoder resolved _device_type and never reported it, so the documented check showed a single line and looked like the reranker was still on CPU. Log the device on both reranker init paths and show the real log output in the README. - Dropped the hardcoded `--platform=linux/amd64`. PyTorch ships cu126 wheels for aarch64 too, and on an arm64 host the pin silently produced an emulated amd64 image that cannot reach the GPU at all. Documented instead that the build must be native. - Dropped `--index-strategy unsafe-best-match`. It relaxed the index isolation that hindsight-api-slim/pyproject.toml deliberately sets up, and it was not needed: resolving without it succeeds and yields the same package set. - Documented the image size (~11 GB vs ~9 GB for the base) in installation.md and the recipe README, since that cost is the reason no CUDA image is published. - Added a HINDSIGHT_VERSION build arg so the base tag can be pinned, fixed the manual `docker build` context, and enabled RERANKER_LOCAL_FP16 in the compose file (faster on GPU, quality-identical). Verified: `uv pip install` inside the base image replaces only torch (2.10.0+cpu -> 2.10.0+cu126) and adds the nvidia/cuda runtime wheels; compose config validates; docs skill regenerates clean; test_local_cross_encoder.py passes (21). Claude-Session: https://claude.ai/code/session_017ufCz6qrNxn36Stug7ek8A --------- Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
39 lines
2.0 KiB
Docker
39 lines
2.0 KiB
Docker
# Example: build a custom Hindsight image with CUDA-enabled PyTorch
|
|
# for NVIDIA GPU-accelerated local embedding and reranker models.
|
|
#
|
|
# Based on the full standalone image (which already bakes in the default
|
|
# BAAI/bge-small-en-v1.5 and cross-encoder/ms-marco-MiniLM-L-6-v2 models).
|
|
# The CPU torch wheel stays in the base layers, so the CUDA runtime is purely
|
|
# additive: expect roughly 11 GB on disk against ~9 GB for the base image.
|
|
#
|
|
# Requirements on the host:
|
|
# - NVIDIA GPU with driver >= 525.60.13
|
|
# - NVIDIA Container Toolkit (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
|
|
# - Docker daemon configured with nvidia runtime (or run with --gpus all)
|
|
#
|
|
# PyTorch publishes CUDA wheels for both x86_64 and aarch64, so this builds on
|
|
# either architecture. Build natively on the host that will run it — an emulated
|
|
# cross-architecture image cannot reach the GPU.
|
|
|
|
ARG HINDSIGHT_VERSION=latest
|
|
FROM ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION}
|
|
|
|
ARG PYTORCH_CUDA_FLAVOR=cu126
|
|
ARG PYTORCH_CUDA_INDEX=https://download.pytorch.org/whl/cu126
|
|
|
|
# Upgrade PyTorch inside the venv to the CUDA runtime matching the installed base
|
|
# version. uv pip install against the image's venv replaces the CPU torch wheel
|
|
# with torch+cu126 and adds the NVIDIA CUDA 12 runtime libraries (cuDNN, cuBLAS,
|
|
# etc.). Everything already installed that still satisfies the resolution is kept,
|
|
# so torch is the only package replaced.
|
|
RUN TORCH_BASE_VERSION=$(/app/api/.venv/bin/python -c \
|
|
'import torch; print(torch.__version__.split("+", 1)[0])') \
|
|
&& uv pip install \
|
|
--python /app/api/.venv/bin/python \
|
|
--no-cache \
|
|
--index "$PYTORCH_CUDA_INDEX" \
|
|
"torch==${TORCH_BASE_VERSION}+${PYTORCH_CUDA_FLAVOR}" \
|
|
&& uv pip check --python /app/api/.venv/bin/python \
|
|
&& /app/api/.venv/bin/python -c \
|
|
'import torch; assert torch.version.cuda, "CUDA-enabled PyTorch wheel was not installed"; print(f"PyTorch CUDA runtime: {torch.version.cuda}")'
|