Addresses the `ccc init` issues reported in #181:
- Relabel embedding providers so local Ollama is clearly under `litellm`
("litellm (100+ providers — cloud APIs & local Ollama)") and
sentence-transformers is marked as built-in HuggingFace models.
- Reject `ollama/` models inline at the sentence-transformers prompt, before
anything is written or tested, instead of crashing later.
- On a failed init model check, loop with an interactive "try a different
model / keep & finish" choice, pre-filling the previous provider and model
on retry, and print a prominent "Next steps" recovery block.
- Add `ccc doctor -v` to show full tracebacks; by default show the one-line
error plus a hint to rerun with `-v`.
- Fix the retry crash where a rewritten global_settings.yml made the already
-ensured daemon report a bogus "version mismatch": restart the daemon on a
stale-settings handshake even after it was ensured, while still failing fast
on a genuine mid-session version mismatch. Make DaemonVersionError's message
reflect the actual cause.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
When a daemon-side `ccc doctor` check failed (e.g. the model check), only a
one-line, 500-char-truncated summary reached the CLI; the full traceback was
logged to daemon.log but never surfaced, making failures hard to debug.
Carry the full traceback across both daemon→CLI error paths:
- Per-check failures: add `traceback` to `EmbeddingCheckResult` and
`DoctorCheckResult`; `check_embedding` captures `format_exc()` and
`_check_model` propagates it. The CLI prints it dimmed under the error.
- Streaming exceptions: add `traceback` to `ErrorResponse`, populated by the
daemon's streaming handler and appended to the client-raised RuntimeError.
Both new fields default to None, keeping the msgpack wire format
backward-compatible.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Voyage models (e.g. voyage/voyage-code-3) reject encoding_format="float"
and require base64. Only inject encoding_format/drop_params for providers
that accept the float hint, leaving voyage/ and bedrock/ to use their
native defaults.
Fixes#148
- Remove `dimensions` from the litellm whitelist in `_ACCEPTED_KWARGS`.
Output dimension must be identical for indexing and query for vectors to
be comparable, so it's a model-wide setting, not a per-side knob —
exposing it under `indexing_params` / `query_params` invited
misconfiguration. Updated comment template, README, design doc, and
testing plan accordingly.
- Plumb `indexing_params` into `create_embedder` and pass them as
constructor kwargs to `PacedLiteLLMEmbedder`. The values land in
`self._kwargs` and become defaults forwarded into every
`litellm.aembedding` call — including paths that don't go through the
`INDEXING_EMBED_PARAMS` context var (e.g. the dim probe in `_get_dim`).
Per-call overrides (`query_params` spread at query time) still win
because `_embed` overlays kwargs on top of `self._kwargs`. Sentence-
transformers ignores `indexing_params` (its constructor doesn't accept
arbitrary kwargs; `prompt_name` is per-call only).
Users can now set `indexing_params` and `query_params` under `embedding:` in
`global_settings.yml` to pass extra kwargs to the embedder separately for
indexing vs. query — supporting asymmetric retrieval models (Cohere v3,
Voyage, Nvidia NIM, Gemini, nomic-ai code/text models, Snowflake arctic,
etc.).
- `ccc init` auto-populates these from a curated table of known models and
prints the applied defaults; unknown models get a commented-out template
for the accepted keys (`prompt_name` for sentence-transformers;
`input_type`, `dimensions` for litellm).
- Daemon validates the effective params at startup; invalid keys fail fast
with a clear error.
- Backward compat: configs for `nomic-ai/CodeRankEmbed` /
`nomic-ai/nomic-embed-code` that predate this feature keep the previous
hardcoded `prompt_name=query` behavior, and a one-time handshake warning
asks users to make the setting explicit. The warning is suppressible by
any non-None `query_params` (including `{}`).
- `ccc doctor` now tests indexing and query separately so asymmetric
misconfigurations surface independently.
Drops the legacy `shared.query_prompt_name` module variable and
`_QUERY_PROMPT_MODELS` set; the new resolution path is centralized in
`embedder_params.resolve_embedder_params` and the curated defaults live in
`embedder_defaults._DEFAULT_PARAMS`.
Also enables `litellm.drop_params = True` so provider-specific kwargs that
a particular model doesn't accept are silently dropped instead of failing.
The module-level `shared.embedder` global was written by `create_embedder()`
but never read from production code — the embedder flows through the
`EMBEDDER` ContextKey via `context.provide` / `use_context`.
Drop the global along with the two test readers that kept it alive:
- `tests/test_daemon.py` — the `daemon_sock` fixture pre-loaded an embedder
and monkeypatched `dm.create_embedder` to reuse it. That was only useful
as a cross-module cache via the now-dead global; with a session-scoped
fixture, `run_daemon()` loads the embedder once from the saved settings
regardless.
- `tests/test_chunker_registry.py` — removed a stale
`monkeypatch.setattr(_shared, "embedder", stub)` whose comment claimed
CodeChunk.embedding read the global at schema resolution time. It reads
the EMBEDDER ContextKey instead; the stub is already wired through
`Project.create(..., stub, ...)`.
LiteLLM documents encoding_format as defaulting to float, but in
practice some providers (e.g. nvidia_nim) error out when it's not
provided. Pass it explicitly as a workaround.
* perf(docker): split install into stable deps + per-release layers; add GHA cache
Dockerfile previously installed cocoindex-code, cocoindex, torch,
sentence-transformers, and all transitive deps in one RUN. Any change to
the source tree (via COPY . /ccc-src) invalidated that single layer,
forcing a full re-install — ~1 GB of wheels for torch + friends — on
every release. Under QEMU for the arm64 cross-build this was slow
enough to be painful.
Split into two stages:
- `deps`: install cocoindex + cocoindex-code[default] from PyPI. Cache
key is just the RUN command string, so this layer is reused across
releases until we bump the pins.
- `builder`: overlay the release version via
`CCC_INSTALL_SPEC=/ccc-src[default]` with `--no-deps
--force-reinstall` — only the cocoindex-code package is touched; the
heavy deps layer stays untouched.
Also add BuildKit layer cache (`type=gha`) to the publish-docker job so
the deps layer persists across workflow runs, not just within a single
build.
* feat(docker,packaging): slim/full image variants; rename [default]→[full] extra
Build two Docker image variants per release:
- slim (:latest, default) — ~450 MB. LiteLLM-only. cocoindex + cocoindex-code
without sentence-transformers. Targets cloud-backed embeddings.
- full (:full) — ~5 GB. Bundles sentence-transformers + torch +
a pre-baked default model. Targets offline-ready local embeddings.
Dockerfile gains a CCC_VARIANT build arg that gates stage 1's
sentence-transformers install and stage 3's model bake. Release workflow
matrices on {slim, full}; each variant has its own GHA cache scope so
layer reuse works across releases without the variants evicting each
other.
Also rename the PyPI `[default]` umbrella extra to `[full]` so pip and
Docker names match. `[embeddings-local]` remains the canonical primary
extra (the one that specifically pulls in sentence-transformers); `[full]`
is its umbrella alias that may bundle additional optional niceties later.
CLI hints that point at missing sentence-transformers continue to name
`[embeddings-local]` directly — the most specific pointer for that case.
README documents both image variants with a comparison table and narrows
the Mac-on-Docker MPS note to only :full users (slim + LiteLLM is
unaffected).
* feat: unified Docker workspace mount with supervised daemon
Reshape the Docker experience around a single bind mount and a single
named volume. Global settings live on the host under
$HOME/.cocoindex_code/ (visible and editable); index data and the model
cache persist in one cocoindex-data volume; daemon runtime state stays
on the container's native filesystem.
CLI and MCP output now show host-side paths via a bidirectional
COCOINDEX_CODE_HOST_PATH_MAPPING translator. A shell wrapper that
forwards $PWD (COCOINDEX_CODE_HOST_CWD) lets ccc work from any project
subdirectory on the host.
The daemon tolerates a missing global_settings.yml (starts in
no-settings mode) so ccc init's interactive picker works in Docker on
first run. A supervisor restart loop in the entrypoint, driven by a new
COCOINDEX_CODE_DAEMON_SUPERVISED contract, makes settings-change
auto-restart safe — editing global_settings.yml triggers an in-place
daemon respawn without taking the container down.
Linux ownership alignment via PUID/PGID, gosu privilege drop, and a
coco user baked into the image. Release workflow now publishes to both
Docker Hub (cocoindex/cocoindex-code) and GHCR
(ghcr.io/cocoindex-io/cocoindex-code).
Also:
- Merge cocoindex-db and cocoindex-model-cache into a single volume
- find_parent_with_marker requires .cocoindex_code/settings.yml, so a
workspace-root global-only dir doesn't trigger nested-init warnings
- New pytest marker `docker_e2e` gates the Docker-backed E2E suite
(excluded from default pytest runs)
* fix: mypy on Windows for POSIX-only os.getuid/getgid calls
- Move `sentence-transformers` behind `[embeddings-local]` and `[default]`
extras (via `cocoindex[sentence-transformers]`), so `pip install
cocoindex-code` is LiteLLM-only. Closes#117.
- `ccc init` is now interactive when global settings don't exist: pick
provider (sentence-transformers / litellm) and model via a
questionary TUI. New `--litellm-model MODEL` flag skips prompts and
is the non-TTY escape hatch for LiteLLM. Closes#70.
- Change the default sentence-transformers model from
`all-MiniLM-L6-v2` to `Snowflake/snowflake-arctic-embed-xs`
(lighter, better quality for code).
- Generated `global_settings.yml` now includes a `ccc doctor` reminder
and commented-out env-var examples (OPENAI_API_KEY, GEMINI_API_KEY,
ANTHROPIC_API_KEY, VOYAGE_API_KEY).
- Model test during init runs in the daemon via the existing
`DoctorRequest` path; the daemon loads the model once and stays
running, so the user's next `ccc index` starts warm.
- Docker image now installs `cocoindex-code[default]` and pre-caches
the new default model. The `COCOINDEX_CODE_EMBEDDING_MODEL` env var
is no longer documented for Docker; users mount a
`global_settings.yml` or pass `--litellm-model`.
- Extract `check_embedding` + `EmbeddingCheckResult` into `shared.py`;
refactor daemon `_check_model` to delegate. Error messages in doctor
output now include the exception type name (strictly more
informative).
- Tests switch to a lighter `paraphrase-MiniLM-L3-v2` model via a new
`make_test_user_settings()` helper in `conftest.py`, leaving CI
cache costs unchanged.
* refactor: extract daemon path helpers to avoid CLI importing cocoindex
Move daemon_dir, daemon_socket_path, daemon_pid_path, daemon_log_path,
and connection_family from daemon.py into a new lightweight _daemon_paths.py
module. This prevents the CLI client from transitively importing cocoindex
and its heavy dependencies (numpy, torch, etc.) when it only needs path
utilities.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: mark free-threaded Python CI jobs as continue-on-error
tokenizers lacks pre-built wheels for cp314t (free-threaded ABI),
so uv falls back to compiling from Rust source. This source build
is fragile across macOS runner image updates and broke after the
20260406 image bump. Mark 3.14t jobs as continue-on-error since
free-threaded wheel coverage across the ecosystem is still limited.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: enable mypy explicit_package_bases, remove stale type: ignore comments
explicit_package_bases = true resolves module paths from the repo root,
preventing double-discovery of files in tests/ under different module names.
Required for tests/example_toml_chunker.py to be importable as
example_toml_chunker rather than an ambiguous bare module.
Also sets asyncio_mode = auto so pytest-asyncio behaviour is explicit
(default in 1.3.0 is strict).
With explicit_package_bases active, mypy can fully resolve the Response
union in test_daemon.py — the 16 type: ignore[union-attr] and
type: ignore[attr-defined] comments that were suppressing false positives
under the old resolution are now unused and removed.
# Conflicts:
# tests/test_daemon.py
* feat: pluggable chunker registry with settings integration
Improves retrieval precision by letting users split specific file types at
semantic boundaries (e.g. TOML sections, SQL statements) instead of the
default line-window splitter.
## What
- cocoindex_code/chunking.py: public API module exporting ChunkerFn
(Callable alias), CHUNKER_REGISTRY context key, and re-exports of
Chunk/TextPosition from upstream. Single import path for chunker authors.
- tests/example_toml_chunker.py: demo chunker splitting at [section]
headers; excludes [[array_of_tables]] via negative lookahead. Lives in
tests/ to signal it belongs to a separate package, not the core library.
- ProjectSettings.chunkers: new list[ChunkerMapping] field, serialised as
YAML. Each entry maps a file extension to a 'module.path:callable' string.
Users activate chunkers by editing .cocoindex_code/settings.yml — no code
changes required.
- daemon.py: _resolve_chunker_registry resolves ChunkerMapping entries via
importlib at project load time and passes the result to Project.create().
callable() guard gives a clear error at startup rather than a TypeError
per file.
- Project.create(chunker_registry=...): new optional parameter. Injected as
a cocoindex context key (tracked=False) rather than exposed via env internals.
Empty registry by default — zero behavioural delta for existing users.
- indexer.py: process_file checks the registry per file suffix; falls through
to RecursiveSplitter unchanged when no chunker is registered.
## Design decisions
- ChunkerFn returns (language_override, chunks): language_override=None keeps
detect_code_language() result; non-None lets the chunker correct it (e.g.
.sls files starting with #!py).
- tracked=False is consistent with SQLITE_DB, CODEBASE_DIR, and other
non-serialisable context keys. Changing a chunker requires a daemon restart,
which triggers a full re-index anyway.
- _resolve_chunker_registry lives in daemon.py, its only call site, keeping
settings.py as pure schema/IO and chunking.py as pure type definitions.
# Conflicts:
# src/cocoindex_code/daemon.py
# src/cocoindex_code/indexer.py
# src/cocoindex_code/project.py
# tests/test_settings.py
Add target_sqlite_db_path() and cocoindex_db_path() to settings.py as
the single source of truth for database paths, replacing scattered
hardcoded "target_sqlite.db" and "cocoindex.db" strings across cli.py,
daemon.py, project.py, and config.py. Also use daemon_log_path() for
daemon log references and project_settings_path() for settings file
references.
Remove config.py (unused legacy module) and its tests — no production
code imported it.
Add settings/index-db location display to `ccc status`.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add COCOINDEX_CODE_DB_PATH_MAPPING for custom database locations
Allow remapping database file locations via environment variable, enabling
Docker deployments where databases live on the container's native filesystem
while source code is mounted from the host.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use = delimiter for DB path mapping (avoid Windows colon conflict)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: use platform-agnostic tmp_path in DB mapping tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: revisit and simplify CLI-daemon connection model
- CLI-daemon connections are stateless now.
- Also stop caching project settings in memory. Much easier to pick up
new version.
* fix: type annotation
Background indexing during MCP startup shared the same DaemonClient
(multiprocessing connection) as foreground search requests. Concurrent
sends/recvs on the single pipe corrupted data, causing:
Query failed: Input data was truncated
_bg_index() now acquires a fresh DaemonClient via ensure_daemon(),
uses it for indexing, and always closes it in finally. Both MCP
startup paths are fixed (cli.py `ccc mcp` and server.py `cocoindex-code`).
* fix: make sure daemons are brought down cleanly
* chore: be more verbose in pre-commit workflow
* fix: split CI workflow to avoid prek output capture deadlock on Windows
prek --verbose captures subprocess stdout via pipe. On Windows, the 4KB
pipe buffer fills when pytest produces verbose output, causing a deadlock
(pytest blocks writing, prek blocks waiting for exit). Split into two
steps: prek --skip pytest for lint checks, and pytest running directly
for streaming output.
Also:
- Add OSError catch in _pid_alive for Windows edge cases
- Properly shut down daemon thread in test_daemon.py session fixture
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: catch SystemError from os.kill on Windows
On Windows, os.kill(pid, 0) raises SystemError when the target process
has exited but its handle state is in transition (WinError 87). This
corrupts CPython C exception state, causing subsequent built-in calls
like time.monotonic() to also raise SystemError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: avoid os.kill on Windows due to CPython C exception state corruption
os.kill(pid, 0) on Windows corrupts CPython C-level exception state
even after the raised OSError is caught. This causes subsequent calls to
C built-ins (time.monotonic, time.sleep) to raise SystemError. Use
ctypes OpenProcess instead, which is the proper Win32 API for checking
process existence.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: stablize setting files
* fix: improve daemon readiness detection on Windows
On Windows, os.path.exists() is unreliable for named pipes (\\.\pipe\*).
Replace it with actual connection attempts in _wait_for_daemon and
is_daemon_running. Also increase the default timeout from 10s to 30s
since Windows CI can be slower during model loading.
Additionally, after TerminateProcess on Windows (SIGTERM fallback in
stop_daemon), wait for the old process to fully exit before returning,
so that named pipe handles are released before starting a new daemon.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes#62
- add pathspec dependency and load project .gitignore
- apply gitignore-aware matcher that handles nested rules
- add regression test ensuring ignored files stay out of the index
Co-authored-by: Clawdbot <bot@clawd.bot>
* feat: assorted improvements for CLI/daemon to be resilient and ergonomic
* tests: add more e2e tests
* fix: resolve path comparison and LMDB cleanup issues on Windows/3.14t
Three root causes fixed:
1. Path resolution mismatch on Windows: init and auto_init_project used
unresolved Path.cwd() but find_parent_with_marker/find_project_root
resolve internally, causing comparison failures and daemon key mismatches.
2. LMDB not released on remove_project: The daemon ProjectRegistry
dropped the Project from dicts without closing its SQLite connection or
forcing GC of the Rust LMDB environment. On free-threaded Python (3.14t)
and Windows, deferred GC kept the LMDB open, causing environment already
open errors and PermissionErrors when deleting db files.
3. Silent connection close on streaming errors: When update_index async
iteration failed in the daemon, the connection was closed without sending
an ErrorResponse, causing the client to get an unhelpful EOFError.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: break LMDB reference chain explicitly and resolve cwd in path filter
On 3.14t (free-threaded Python), gc.collect() alone does not release the
Rust LMDB environment because deferred reference counting keeps
core.Environment alive through App._core_env_app, ContextProvider._core_env,
and Environment._core_env. Explicitly null these internal references in
Project.close() before gc.collect() so the Rust object is freed promptly.
Also resolve Path.cwd() in resolve_default_path() — on Windows, the
unresolved cwd did not match the resolved project_root, causing
relative_to() to fail and the subdirectory path filter to be skipped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use double gc.collect() for layered deferred refcount on 3.14t
Revert fragile internal attribute clearing. On free-threaded Python, the
first gc.collect() frees Python wrappers whose Rust Drop implementations
issue further deferred Py_DECREF calls on core.Environment; a second
gc.collect() flushes those and actually drops the LMDB handle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add sleep in tests for LMDB release on free-threaded Python
On 3.14t, deferred reference counting means the Rust LMDB environment
is not released immediately after remove_project + gc.collect(). Add a
1-second sleep in the two tests that reset and re-index, giving the
runtime time to process pending deferred Py_DECREF calls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: restart daemon after reset in tests, increase daemon wait timeout
Instead of sleeping, restart the daemon after reset in the two tests
that re-index after removing databases. This reliably releases the LMDB
environment on all platforms including free-threaded Python (3.14t).
Also increase _wait_for_daemon timeout from 5s to 10s — Windows CI
runners occasionally need longer to start the daemon subprocess.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: allow adding file extensions via COCOINDEX_CODE_EXTRA_EXTENSIONS env var
Users can now specify additional file extensions to index without editing
source code, e.g. COCOINDEX_CODE_EXTRA_EXTENSIONS="inc,tpl"
* feat: support language mapping in COCOINDEX_CODE_EXTRA_EXTENSIONS
Allow `ext:lang` format (e.g. "inc:php,yaml,tpl:html") to override
language detection for extra extensions, enabling proper AST-based
chunking for unrecognized extensions.
- Reverts _DEFAULT_MODEL from CodeRankEmbed back to all-MiniLM-L6-v2
- Adds batch_size: int field to Config dataclass (default 16, env var COCOINDEX_CODE_BATCH_SIZE)
- Adds module-level config singleton for import by shared.py and embedder.py
- TDD: tests written first (3 failing), then implementation made them pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- shared.py: log _trust (effective value) instead of config.trust_remote_code
so the log correctly shows True for CodeRankEmbed regardless of env var
- tests: add memo key test for query_prompt_name dimension
- tests: add embed_query tests asserting prompt_name is forwarded/omitted
correctly (regression guard on asymmetric retrieval behaviour)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- embed_query: use explicit prompt_name= kwarg instead of **kwargs so
mypy can type-check against SentenceTransformer.encode overloads
- query_codebase: remove stale type: ignore (mypy narrows union type
via hasattr, so union-attr error doesn't exist in that branch)
- test: modernize isinstance tuple to str | bool (ruff UP038)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace jina-embeddings-v2-base-code (broken with transformers 5.x) with
nomic-ai/CodeRankEmbed: same 137M params and 8192-token context, but
state-of-the-art code retrieval (outperforms jina by ~10 MRR points) and
fully compatible with transformers 5.x via its own custom NomicBERT code.
Changes:
- Switch default model to sbert/nomic-ai/CodeRankEmbed
- Add query_prompt_name + embed_query() to LocalEmbedder for asymmetric
retrieval (CodeRankEmbed uses prompt_name="query" for queries, no prompt
for indexed code chunks)
- Auto-enable trust_remote_code for known-compatible models (CodeRankEmbed)
- Use embed_query() in query_codebase() instead of embed()
- Reduce max_batch_size 64→16 (prevents OOM with 8192-token attention)
- Add einops dependency (required by CodeRankEmbed custom code)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Jiangzhou