Introduction

semanticastindexer (SAI) is a Rust CLI and MCP server for semantic code search and near-duplicate function detection over any codebase. You index your source once, then you can do two things with it:

• Search in plain English — ask “where do we open the DuckDB connection?” and get the matching code, ranked by meaning rather than keywords.
• Surface near-duplicate functions — find the functions across the repo that are near-copies of each other, so you can de-duplicate or refactor.

SAI works the same whether you drive it yourself from the terminal or wire it into an AI coding agent as an MCP server.

Highlights

• Local-first and offline. Embeddings run on-device through ONNX Runtime (via ort) — no API keys, nothing leaves your machine. A compact code model is pulled once from Hugging Face (jina-embeddings-v2-base-code, or e5-small). If you prefer, you can instead point at an Ollama server over HTTP.
• Pluggable backends. Store vectors in a local DuckDB file (VSS/HNSW, fully offline) or in Qdrant Cloud with server-side inference.
• Symbol-aware AST chunking. Code is split per symbol using tree-sitter for TypeScript/TSX, Rust, Go, and Python; every other language falls back to a line-based chunker.
• You control what leaves the repo. A YAML config filters out tests, generated files, comments, and more, and sai-noindexing / sai-noduplicate opt-out markers let you exclude code inline.
• Read-only by default. The MCP server only searches; the write tool (sai_refresh) requires --allow-write.

Who it is for

• Developers navigating a large or unfamiliar codebase who want to find code by intent instead of grepping for exact names.
• Teams fighting duplication who want to detect copy-pasted or near-identical functions across the whole repo.
• AI coding agents (Claude Code, Cursor, Windsurf, Codex, and others) that need a fast, local, semantic view of the code they are editing.

The mental model

Think of SAI as a search index for meaning, built once and refreshed as code changes:

1. Index. SAI walks your source, filters it, splits it into chunks (per symbol for AST languages, by lines otherwise), embeds each chunk into a vector, and upserts it into your chosen backend. Point IDs are a deterministic hash of path + start_line, so re-running updates points in place instead of duplicating them.
2. Query. Your natural-language question is embedded the same way, and the backend returns the nearest chunks.
3. Duplicates. SAI compares chunk vectors across the codebase and clusters the near-neighbours into near-duplicate groups.

You can expose the same index through the CLI or the MCP server: the shipped tools are sai_search_code, sai_find_similar, sai_find_duplicates, sai_index_status, and sai_refresh.

Next steps

• Getting Started — install SAI, index a project, and run your first search and duplicate scan.
• How It Works — the indexing pipeline, point IDs, and payload shape in detail.
• Glossary — the key terms (chunk, embedder, backend, symbol, point) used throughout the book.

Getting started

This is a single, linear walkthrough: install the binary, index a project, ask it a question in plain English, find near-duplicate functions, and (optionally) connect a coding agent. The whole tutorial uses the fully-offline default — the DuckDB backend with the on-device ort (ONNX Runtime) embedder — so there are no API keys and no servers to set up. After the first model download, nothing else leaves your machine.

If you only want the reference rather than a tutorial, jump to CLI usage or the installation guide.

1. Install

One line, no Rust toolchain required (it downloads a prebuilt binary from the latest GitHub Release and puts semanticastindexer on your PATH):

# macOS / Linux
curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash

# Windows (PowerShell)
powershell -c "irm https://maadgrom.github.io/semanticastindexer/install.ps1 | iex"

Prefer to build it yourself? Build with all features enabled so the offline default (DuckDB + ort) works out of the box, then use the binary at ./target/release/semanticastindexer:

cargo build --release --features all

See the installation guide for build requirements and details.

2. Move into your project

Run the binary from the repo root of the project you want to search, so the stored paths are project-relative:

cd /path/to/your/project

Optionally generate a starter sai-cfg.yml — the fully-commented standard config. A short interview asks for the backend, embedder, collection, and model (Enter accepts every default; --yes skips the questions entirely):

semanticastindexer init

Without a config, the built-in defaults apply — fine for this tutorial. See the configuration reference for every key.

3. Dry-run to preview what gets indexed

Before touching the index, do a dry-run. It reports exactly which files would be included or skipped — no network, no embedding, no model download:

semanticastindexer --root src --ext ts,tsx --dry-run

Expected output (abbreviated — your file list will differ):

[include] src/app.ts
[include] src/utils/format.ts
[skip]    src/app.test.ts        (test file)
[skip]    src/components/ui/...   (shadcn)
...
dry-run: 42 included, 7 excluded (no upload)

--root defaults to src and --ext defaults to ts,tsx. Adjust them for your project (for example --ext go or --ext rs). Always dry-run first to confirm the inclusion set.

4. Index it

Now run the real index. It creates the local DuckDB table if it’s missing and embeds each chunk on-device:

semanticastindexer --root src --ext ts,tsx

First run downloads a model. With the default ort embedder, the code-trained model jina-embeddings-v2-base-code (161M params, 768-dim) is pulled from Hugging Face the first time you index. That’s a few hundred MB and only happens once — it’s cached for every later run. If the download stalls or fails (proxy, offline, Hugging Face hiccup), see troubleshooting.

Expected output (abbreviated):

downloading model jinaai/jina-embeddings-v2-base-code ... done
indexing src (ext: ts,tsx) → collection source_code
embedded 318 chunks
upserted 318 points
done

Subsequent indexes reuse the cached model and start immediately.

5. Ask a question in plain English

Search the index with --query-only (this skips indexing and only searches — it never uploads your codebase):

semanticastindexer --query-only --query "where do we open the duckdb connection"

Expected output (abbreviated — --limit defaults to 5 results):

0.71  src/db/connection.ts:12-40   openDuckDb
0.64  src/db/pool.ts:8-31          createPool
0.58  src/index.ts:55-77           bootstrap
...

Each line is score path:start-end symbol. Higher scores are closer matches.

6. Find near-duplicate functions

The duplicates command scans the stored vectors (no re-embedding) and groups near-identical functions into clusters using nearest-neighbour edges plus union-find:

semanticastindexer duplicates

Expected output (abbreviated):

cluster (size 3, sim 0.94..0.97):
  src/utils/format.ts:10-24      formatDuration
  src/lib/time.ts:31-44          humanizeSeconds
  src/components/Clock.tsx:60-72  toClock

cluster (size 2, sim 0.93..0.93):
  src/api/users.ts:18-29         mapUser
  src/api/admin.ts:40-51         mapAdminUser

Clusters print largest-first. The built-in cutoff is a similarity of 0.93; you can tune it with --min-score, --top-k, and friends. See CLI usage for the full set of duplicates and similar flags.

7. Connect a coding agent (optional)

The binary is a complete CLI on its own, but you can also expose it to a coding agent over MCP. Re-run the installer with --platform claude-code (macOS/Linux) or -Platform claude-code (Windows) and it wires up the project’s .mcp.json (and installs the Claude Code skill):

# macOS / Linux
curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform claude-code

# Windows (the scriptblock form is how flags pass through irm)
powershell -c "& ([scriptblock]::Create((irm https://maadgrom.github.io/semanticastindexer/install.ps1))) -Platform claude-code"

Other supported ids include claude-desktop, cursor, windsurf, continue, codex, hermes, ollama, and generic. See the installation guide for per-platform config locations.

Once your agent has restarted and picked up the MCP server, ask it something that triggers a search. It calls the sai_search_code tool — the same semantic search you ran in step 5:

You: Using semantic code search, where do we create the Qdrant collection?

Agent (via sai_search_code):
  src/db/qdrant.ts:22-48  ensureCollection — creates the collection if it doesn't exist
  src/db/setup.ts:9-30    bootstrapVectorStore

The other shipped tools are sai_find_similar, sai_find_duplicates, sai_index_status, and the write-only sai_refresh (which requires --allow-write). The server is read-only by default.

What next

• Tune what leaves the repo and learn the opt-out markers via the installation guide and CLI reference.
• Hit a snag (model download, no results, wrong paths)? See troubleshooting.

Installation

SAI ships as a single self-contained binary. The quickest path is the one-line installer, which downloads a prebuilt binary from the latest GitHub Release — no Rust toolchain required. You can also wire up a coding agent in the same step, or build from source if there’s no release for your platform.

Quick install (per OS)

# macOS / Linux
curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash

# Windows (PowerShell)
powershell -c "irm https://maadgrom.github.io/semanticastindexer/install.ps1 | iex"

Prefer to pick your OS interactively? Use the hosted install page: maadgrom.github.io/semanticastindexer.

On every OS the installer downloads the binary, then asks which coding agent(s) to connect (reading your keypress straight from the terminal, so the prompt works even under curl | bash). Press Enter to skip the prompt and install only the binary — it’s a full CLI on its own. See the CLI reference to start indexing immediately.

Connect your coding agent

Connecting an agent is optional. Add --platform <id> (macOS/Linux) or -Platform <id> (Windows) and the installer wires up that client’s MCP config (and, for Claude Code, installs the sai skill into ~/.claude/skills/):

# macOS / Linux
curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform cursor

# Windows (the scriptblock form is how flags pass through irm)
powershell -c "& ([scriptblock]::Create((irm https://maadgrom.github.io/semanticastindexer/install.ps1))) -Platform cursor"

Supported ids: claude-code, claude-desktop, cursor, windsurf, continue, codex, hermes, ollama, generic. install.ps1 takes the same flags PowerShell-style: -Platform <id>, -All, -NonInteractive, -Write, -Collection <name>, -Embedder <id>, -SkipBinary.

By default — without --write — the installer prints the config snippet and the exact target file path so you can paste it yourself. The merge with --write only applies to JSON-based clients; for Continue (YAML) and Codex (TOML) the installer always prints the block to paste.

When no client is selected, no tty is available, or you pass --non-interactive, the installer prints a generic MCP block. The generated snippet runs the server as semanticastindexer mcp --backend duckdb --embedder <id> --collection <name> with cwd set to the directory you ran the installer from.

For the full per-client walkthrough, see MCP clients.

Per-platform config locations

The MCP server entry is always registered under the name sai. For the tool surface it exposes — sai_search_code, sai_find_similar, sai_find_duplicates, sai_index_status, sai_prepare_mcp_setup, and sai_refresh — see the MCP server and tools reference.

Embeddings

The DuckDB backend embeds locally via a pluggable embedder (embedder: ort | ollama):

• ort (default) — on-device ONNX Runtime. No server, no API keys. The model is pulled from Hugging Face on first run: the code-trained jina-embeddings-v2-base-code (161M params, 768-dim), or intfloat/multilingual-e5-small (118M, 384-dim) as the zero-config text default. Swap in any ONNX embedding model on Hugging Face by setting model plus a matching vector_dim.
• ollama — embedding server over HTTP. Point at a local or remote Ollama server. Handy in CI/CD, where an embedding service often already runs: ollama serve, ollama pull mxbai-embed-large, set ollama.url + ollama.model, and index. Browse embedding models on Ollama.

See backends and embedders for the full matrix and choosing a model for the recommended code model for de-duplication.

Build from source

If you prefer to build the binary yourself (or there’s no release for your platform), build with all features enabled:

# Recommended — full-featured binary (everything included)
cargo build --release --features all

# Also fine (equivalent)
cargo build --release --features "qdrant,ort,ollama,ast,mcp"

The binary lands at ./target/release/semanticastindexer. The first build is slower because --features all pulls in native dependencies (bundled DuckDB + ONNX Runtime via ort). Subsequent builds are fast thanks to cargo’s incremental compilation.

Requirements: Rust stable toolchain (edition 2024, MSRV 1.88). A rust-toolchain.toml pins stable, so rustup auto-activates the latest stable when you build in this repo.

Then run the one-command setup script to register the MCP server:

./mcp-setup/setup.sh --non-interactive --backend duckdb --embedder ort

This builds the binary, writes sai-cfg.yml, and installs the sai + sai-deslop skills (and the dedup-auditor subagent). Add --platform <id> (claude-code, cursor, windsurf, continue, codex, …) and --write to also wire that client’s MCP config automatically — the same wiring the one-line install.sh does. See MCP clients.

Security

• The Qdrant API key is read only from the QDRANT_API_KEY environment variable (a secret — never commit it). The cluster URL can be set in sai-cfg.yml (qdrant.url) or via QDRANT_URL.
• If an API key is ever exposed, rotate it in the cluster’s API Keys tab.
• Add target/ to .gitignore (build artifact).
• The MCP server is read-only by default; the write tool (sai_refresh) requires --allow-write.

See security and privacy for the full threat model and the environment variables reference for every credential SAI reads.

Uninstall

curl -fsSL https://maadgrom.github.io/semanticastindexer/uninstall.sh | bash

The uninstaller reverses what install.sh did. Pass --yes (or -y) to skip the confirmation prompt; in a non-interactive shell (CI) it proceeds without asking.

Removed:

• The semanticastindexer binary from ~/.cargo/bin and ~/.local/bin (plus any sai wrapper alongside it).
• The Claude Code skill directory ~/.claude/skills/sai/.
• The sai entry from known JSON MCP configs — Claude Desktop, Cursor, Windsurf, and the project’s ./.mcp.json — each backed up to <file>.bak before editing.

Left untouched (delete by hand if you want them gone):

• Per-project index files (.index/) and any sai-cfg.yml.
• The Codex (~/.codex/config.toml) and Continue (~/.continue/config.yaml) entries.
• Any PATH line the installer added to your shell rc (~/.zshrc, ~/.bashrc, ~/.profile).

If you’re troubleshooting a stale config after reinstalling, see the glossary for the meaning of each backend and embedder key.

How it works

SAI (semanticastindexer) turns a codebase into a vector index of code chunks, then answers semantic queries and near-duplicate scans against that index. This page traces the two data paths end to end — indexing (source files → stored vectors) and search (a query → ranked hits) — and explains the worker-thread actor model the MCP server uses to drive a !Send backend safely.

For the user-facing surface, see the CLI reference and the MCP server and tools page. Terms in bold are defined in the glossary.

The indexing pipeline

Indexing walks the project root and transforms each surviving file into embeddable chunks:

walk --root → prune dirs → ext filter → include allow-list → exclude globs
            → generated-marker skip → strip comments
            → chunk (lines: ~60 lines / max_chunk_chars, 8-line overlap; or ast: per symbol)
            → Document("passage: <code>", model) → upsert (batch, server-side embed)

The pure chunking paths (collect_chunks, build_chunks, dry_run) never touch a vector backend — they only decide what to embed. A single bridge function, reindex_file, performs storage I/O (delete + upsert) for one file, and it is shared by both CLI sync and the MCP sai_refresh tool.

Per-file flow in detail

For each file under --root (after directory pruning and the extension filter), the single source-of-truth decision function load_file_for_indexing applies, in order:

1. include (allow-list) — if the include set is active, the file must match an include glob, else it is skipped.
2. exclude globs — if matched, the file is skipped (exclude always wins over include). Include and exclude are evaluated together by passes_globs.
3. skip_generated — the first 600 bytes are scanned (case-insensitively) for an autogenerated marker (do not edit, @generated, code generated, auto-generated, autogenerated, this file is generated); matches are skipped.
4. The surviving content is read as UTF-8. Binary / non-UTF-8 files are silently ignored.

Then build_chunks produces the embeddable chunks:

1. strip_comments (for C-family extensions) removes // line and /* */ block comments before chunking, so only code is embedded. The stripper preserves string, char, and template literals — and preserves the exact line count, so a chunk’s start_line/end_line still point at the real lines in the original file.
2. The surviving content is split into chunks (see chunking).
3. Each chunk’s raw (pre-strip) line span is scanned for opt-out markers (sai-noindexing drops the chunk entirely; sai-noduplicate flags it so it is stored but excluded from clustering). See opt-out markers.
4. Each surviving chunk is embedded and upserted into the configured backend.

C-family extensions covered by comment stripping are: ts, tsx, js, jsx, mjs, cjs, go, rs, java, c, h, cc, cpp, hpp, cs, kt, swift, scala.

Chunkers

The AST index is deliberately function-only: non-function code (imports, classes, interfaces, type aliases, plain consts, bare closures) is never emitted, so a file with no functions produces no chunks. Selecting chunker: ast on a binary built without the ast feature is a hard startup error. Full details live in chunking.

Deterministic point IDs

Every chunk’s storage ID is a deterministic hash of path + start_line:

#![allow(unused)]
fn main() {
pub fn point_id(path: &str, start_line: usize) -> u64 {
    let mut hasher = XxHash64::with_seed(0);
    path.hash(&mut hasher);
    start_line.hash(&mut hasher);
    hasher.finish()
}
}

XxHash64 with a fixed seed = 0 is stable across builds and machines (unlike DefaultHasher, whose algorithm is unspecified). Because the ID is a pure function of path + start_line, re-running the indexer updates points in place instead of duplicating them, and every backend keys the same chunk identically — so Qdrant and DuckDB agree on IDs.

Payload shape

Each stored point carries this payload:

Chunks also carry a no_duplicate flag (set by the sai-noduplicate marker) and git context (commit_sha, dirty) stamped at index/refresh time.

The search path

All semantic queries embed the input, run a nearest-neighbour (NN) search, then post-filter the over-fetched results. Three shaped reads back the shipped MCP tools and their CLI twins:

• Query search (sai_search_code) — embed text as a query vector, NN-search, return hits. The backend over-fetches (fetch) so a score cut can be applied afterwards. When the backend cannot embed locally (Qdrant), the worker falls back to the backend’s server-side text query() path.
• Similar search (sai_find_similar) — resolve the target, NN-search, then drop everything below min_score:
  • A code snippet is embedded as a passage (code-vs-code space) and searched by that vector.
  • A location (path + 1-based line) reuses the stored chunk’s exact vector — no re-embed — and excludes the chunk itself from its own results.
• Duplicates scan (sai_find_duplicates) — fetch every stored chunk (optionally path-glob filtered), gather each chunk’s top_k nearest neighbours (self-excluded, stored vectors — no re-embed), then run the pure clustering core.

Near-duplicate clustering

cluster_duplicates is a pure union-find over per-chunk neighbour lists, shared by the CLI and the MCP server so the algorithm exists in exactly one place. An edge is kept when its similarity >= min_score; kept edges union the two chunks. Chunks flagged no_duplicate form no edges — neither as a seed nor as a neighbour. Connected components with >= min_cluster_size members become clusters, sorted largest-first (tie-break: higher max_sim) and truncated to max_clusters. Each cluster reports its members plus the min/max edge similarity within it. See tuning similarity for choosing thresholds, and search and duplicates for usage.

The backend worker thread (actor model)

The DuckDB backend embeds a duckdb::Connection (and, with the ort embedder, an ONNX session) that are !Send/!Sync. But the MCP framework (rmcp) requires every tool handler future to be Send, so holding such a backend across an .await inside a tool handler would not compile under the mcp feature.

SAI solves this with an actor / worker-thread pattern:

  MCP server (multi-thread runtime)            backend worker (one OS thread)
  ┌───────────────────────────┐                ┌─────────────────────────────┐
  │ #[tool] handler            │  Request +     │ current-thread Tokio runtime │
  │   BackendHandle (Send+Sync)│  oneshot reply │   owns the !Send Backend     │
  │     mpsc::Sender ──────────┼───────────────▶│   worker_loop: one at a time │
  │   .await oneshot ◀─────────┼────────────────┤   handles each fully, replies│
  └───────────────────────────┘                └─────────────────────────────┘

• A dedicated OS thread (semanticastindexer-backend) owns the Backend and the Plan. It runs its own current-thread Tokio runtime so it can drive the backend’s async methods (e.g. the Ollama embedder’s reqwest calls) locally. The !Send backend never crosses a thread boundary.
• The MCP server holds only a BackendHandle — a Send + Sync wrapper around an mpsc::Sender<Request>. Channels are Send + Sync, so each tool handler captures only Send types and compiles.
• Every tool call builds a Request plus a oneshot reply channel, sends it, and .awaits the reply. The worker loop processes requests one at a time — each request is fully handled (including its .awaits) before the next is taken — so the single DuckDB connection is never touched concurrently. The request channel is bounded (buffer of 32), which is plenty for a strictly sequential consumer.
• The pattern is backend-agnostic: Qdrant’s backend is already Send, but routing every backend through the same worker keeps the handler code identical.

Dropping every BackendHandle clone closes the channel, which ends the worker loop and the thread.

How sync and refresh bracket HNSW bulk windows

DuckDB’s vector index (HNSW) requires bulk mutations to be bracketed so the index is dropped before mass upserts and rebuilt afterwards. Both write paths honour this contract:

• The shared per-file step reindex_file always deletes the file’s existing points first, then — if the on-disk path is still indexable — re-chunks, embeds, and upserts the fresh chunks. The caller is responsible for wrapping the whole operation in a single begin_bulk() / end_bulk() window.
• The MCP sai_refresh tool’s handle_refresh mirrors sync’s correctness contract: begin_bulk (drop HNSW) → per-path delete + re-chunk + re-embed + upsert → end_bulk (rebuild HNSW). A fresh git context is captured per path so the commit_sha/dirty stamp is accurate on a long-lived server.

There is a logical invariant here: end_bulk is always called, even when a path fails mid-batch, so the HNSW index is never left dropped after a refresh. sai_refresh is the single write path exposed by the MCP server and is gated by --allow-write at the call site. See keeping the index in sync and the MCP server reference.

Invariants

A few algorithmic invariants are load-bearing and enforced in the code itself: chunking “nothing dropped” (the dry-run, full-index, and refresh paths share one per-file routine), the DuckDB HNSW bulk contract (begin_bulk/end_bulk around every write), stable cross-backend point IDs, one resolved prefix style applied everywhere, the !Send worker boundary, and the runtime dimension guards. Re-read the relevant modules (src/indexer.rs, src/vectordbs/, src/worker.rs, src/search.rs) before changing core indexing, clustering, or backend logic.

Glossary

Plain-language definitions for the domain terms used across the SAI docs. Terms with a dedicated reference page link to it.

AST / tree-sitter

An Abstract Syntax Tree (AST) is the structured, parsed form of source code. SAI’s ast chunker uses tree-sitter to parse TypeScript/TSX, Rust, Go, and Python and emit one chunk per named function. It is feature-gated behind --features ast; non-function code (classes, types, imports, top-level statements) is deliberately not embedded. See Chunking.

chunk

One embeddable slice of a source file, ready to be turned into a vector and stored. A chunk carries its file path, language, start/end line, text, an optional captured symbol name, the commit it was indexed at, and a dirty flag. The ast chunker emits one chunk per function; the lines chunker emits line windows. See Chunking.

collection

The storage unit on the qdrant backend that holds all of a project’s vectors. Creating the collection (or the equivalent DuckDB table plus index) is what ensure_ready does before indexing. The DuckDB backend uses a local file plus a VSS/HNSW index in place of a remote collection. See Backends & embedders.

cosine similarity

A measure of how close two vectors point in the same direction, ranging up to 1.0 for identical direction. SAI ranks search results and clusters near-duplicates by cosine similarity; because vectors are L2-normalized, cosine similarity is what the DuckDB VSS/HNSW index and Qdrant compare on.

embedding

The numeric vector representation of a chunk’s text produced by an embedder. Similar code produces similar embeddings, which is what makes semantic search and near-duplicate detection possible. The DuckDB backend produces embeddings locally (via ort or ollama); the qdrant backend produces them server-side. See Backends & embedders.

HNSW

Hierarchical Navigable Small World, the approximate-nearest-neighbor graph index used by the DuckDB VSS extension to make cosine search fast. HNSW loses recall after in-place deletes, so a DuckDB sync drops and recreates the index around its changed-file loop — effectively a full graph rebuild. See Backends & embedders.

L2-normalization

Scaling a vector so its length (L2 norm) equals 1, leaving only its direction. The ort pipeline L2-normalizes every embedding after mean-pooling, which lets cosine similarity be compared directly and consistently across all stored vectors. See Backends & embedders.

mean-pooling

Averaging the per-token output vectors of a transformer into a single fixed-size vector. The ort pipeline mean-pools the ONNX last_hidden_state over the attention mask (so padding tokens are excluded) before L2-normalizing — yielding 384 dimensions for e5-small. See Backends & embedders.

MSRV

Minimum Supported Rust Version — the oldest Rust toolchain version the project compiles and is tested against. Building with an older toolchain is unsupported.

near-duplicate cluster

A group of functions whose vectors are mutually close enough (above the duplicate score threshold) to be flagged as near-identical. find_duplicates (CLI duplicates) builds these clusters codebase-wide from stored vectors. Functions marked with a sai-noduplicate marker are still indexed and searchable but excluded from clustering. See union-find clustering.

ONNX Runtime (ort)

The local embedding path used by the DuckDB backend. The ort embedder runs an ONNX Runtime (ort 2.x) inference session over a model and tokenizer downloaded from Hugging Face (onnx/model.onnx + tokenizer.json). Its pipeline is: prefix → tokenize (pad/truncate to 512) → ONNX last_hidden_state → mean-pool → L2-normalize. It is the default embedder. See Backends & embedders.

passage vs query prefix (E5 / Qwen / none)

Asymmetric text prefixes some embedding models expect. The resolved prefix_style (E5, Qwen, or None) is applied by both local embedders and the Qdrant path through one shared helper:

The style is set explicitly via prefix_style or auto-detected from the model name (contains e5 → E5, contains qwen → Qwen, otherwise None). Symmetric code models use none. See Chunking → prefixes.

point ID

The numeric identifier (id, a u64) attached to each stored chunk/vector so it can be located, deduplicated, and self-excluded during search. Raw-vector search over-fetches and dedups by id because HNSW can return the same id more than once; find_similar and find_duplicates use the id to exclude a query chunk from its own results.

semantic vs lexical search

Lexical search matches literal tokens or substrings. Semantic search matches meaning by comparing embeddings, so it can find code that does the same thing with different names or wording. SAI’s sai_search_code is semantic. See the MCP tools reference.

server-side inference

Embedding performed by the remote service rather than locally. The qdrant backend uses Qdrant Cloud’s server-side inference (the Document API) and has no local model, so the server itself turns text into vectors; plain OSS/local Qdrant has no inference engine. Contrast with the DuckDB backend’s local ort/ollama embedders. See Backends & embedders.

symbol

The captured name of a function stored on a chunk by the ast chunker (free functions, methods, and arrow/function-expression consts in TS; functions, impl/trait methods, and nested functions in Rust; func declarations and receiver methods in Go). It is None for line-window chunks, and is surfaced by the similar/duplicates CLI subcommands and by the sai_search_code / sai_find_duplicates MCP tools. See Chunking.

union-find clustering

The disjoint-set algorithm used to merge near-duplicate pairs into clusters: each function that is similar enough to another is unioned into the same group, so a chain of pairwise matches collapses into one cluster. This is how sai_find_duplicates turns pairwise similarity into near-duplicate clusters.

vector

The list of floating-point numbers (Vec<f32>) that represents a chunk’s embedding. Vectors are what the backend stores, indexes (HNSW), and compares by cosine similarity. Search-by-vector reuses an exact stored vector when possible (e.g. find_similar reuses an existing function’s vector rather than re-embedding).

vector_dim

The configured dimensionality of the vectors a backend stores. It must match the embedder model and is validated at runtime — a mismatch is a clear error (embedder produced 768-d vectors but vector_dim=384 …). Examples: e5-small = 384, nomic-embed-text = 768, mxbai-embed-large = 1024. Changing vector_dim requires a fresh index (delete .index/code.duckdb or run with --recreate). See Backends & embedders.

VSS

Vector Similarity Search, the DuckDB extension that provides the HNSW cosine index for the duckdb backend. It must be available for local vector search; the duckdb backend stores vectors in a DuckDB file and queries them through the VSS HNSW index. See Backends & embedders.

Xet storage

Hugging Face’s content-addressed storage system, used by some model repos. It matters because the pinned hf-hub (0.3) fails to fetch tokenizer.json from Xet-backed repos (e.g. jinaai/jina-embeddings-v2-base-code), producing a relative URL without a base error; the workaround is to stage tokenizer.json into the HF cache once with curl until hf-hub is upgraded. See Backends & embedders.

Indexing a project

Building an index is how SAI turns your source tree into searchable vectors. This guide walks through the full task: choosing what to walk, previewing the selection, running the real index, and reading the result. Run the binary from the target project’s repo root so the stored paths are project-relative (or point --root at the project’s source dir).

For the keys referenced here, see Configuration. For how a file becomes chunks, see Chunking. To exempt code from indexing, see Opt-out markers.

What to walk: --root and --ext

Two flags decide which files are even considered:

• --root <dir> — the directory to walk. Default: src.
• --ext <list> — comma-separated extensions, no dots. Default: ts,tsx.

BIN="$(pwd)/target/release/semanticastindexer"   # absolute path to the built binary

# Move into the project you want to index (so payload paths are project-relative).
cd /path/to/your/project

# Index the TypeScript tree.
"$BIN" --root src --ext ts,tsx --collection source_code

Only files whose extension appears in --ext are read. Both flags override sai-cfg.yml.

Dry-run first

Always preview the selection before a real index. --dry-run walks the tree and reports exactly which files would be indexed and which are excluded (and why) — no network, no upload, no quota used:

"$BIN" --root src --ext ts,tsx --dry-run

The report prints the resolved root/ext/collection/model, the active strip_comments / skip_generated_marker settings, the pruned directory names, a WOULD INDEX / EXCLUDED count, a per-reason breakdown (glob, not-included, generated-marker), and a sample of included and excluded paths. The dry-run uses the same shared decision function as the real index, so what it reports is what you get.

Re-indexing: --recreate

By default, indexing creates the collection/table if missing and upserts chunks in place (re-running updates existing points, because each point ID is a stable hash of path + start_line). Pass --recreate to drop and recreate the collection before indexing — a clean slate:

"$BIN" --root src --ext ts,tsx --recreate

A one-time re-index is required for collections built before point IDs became a stable XxHash64(seed=0) of path + start_line. Run "$BIN" flush or index once with --recreate so stale points don’t linger.

Selection order

For each file under --root that survives directory pruning and the --ext filter, the include/exclude decision runs in this exact order:

1. include allow-list — if include is non-empty, the file must match one of its globs, otherwise it is skipped (reported as not-included).
2. exclude globs — if the path matches an exclude glob, it is skipped. Exclude always wins over include.
3. Hard-pruned directories — certain directory names are pruned during the walk regardless of config: node_modules, .git, dist, build, target, .next, coverage, .turbo. Names in exclude_dirs are pruned too.
4. skip_generated_marker — when enabled (default true), the first ~600 bytes of the surviving file are scanned for autogenerated markers (@generated, DO NOT EDIT, code generated, auto-generated, autogenerated, this file is generated). A match skips the file, catching generated files that don’t follow a naming convention.
5. strip_comments — when enabled (default true), C-family // and /* */ comments are removed before embedding so only code reaches the backend. String/template literals are preserved and line numbers stay accurate.

Steps 1–2 are the glob gate; the hard-pruned dirs (step 3) are applied as the walk descends, before any file is even examined.

A minimal sai-cfg.yml controlling these:

include: []                       # empty → consider all files; non-empty → allow-list only
exclude:
  - "**/*.test.ts"
  - "**/*.d.ts"
  - "**/components/ui/**"          # shadcn primitives
  - "**/*.pb.go"                   # Go autogenerated
exclude_dirs:
  - __tests__                      # extra dir names to prune (beyond the hard-coded set)
skip_generated_marker: true
strip_comments: true

Per-extension language labels

Each chunk is stamped with a language payload label derived per file from its extension, lowercased: .ts → ts, .tsx → tsx, Bar.TSX → tsx. So a single --ext ts,tsx walk labels each file with its own language, and you can later filter search or duplicate scans by that label.

Reading the result

A successful index ends with one summary line on stdout:

indexed 1843 chunks from 211 ts/tsx file(s) into 'source_code' (37 file(s) skipped by config)

Reading it left to right: the chunk count, the file count, the extensions (--ext joined by /), the target collection, and (N file(s) skipped by config) — files dropped by the glob gate or the generated-marker scan. (Binary / non-UTF-8 files are silently ignored and are not counted as skipped.) During embedding, progress is printed to stderr; only this final line goes to stdout.

Indexing more languages into the same collection

You can index additional trees into an existing collection by re-running with a different --root / --ext. New points are added; existing ones are updated in place:

"$BIN" --root path/to/go --ext go --collection source_code

Next steps

• Search and find duplicates over what you indexed: Search and duplicates.
• Keep the index current as code changes: Keeping in sync.
• Tune what gets chunked: Chunking.
• Exempt functions from indexing or duplicate clustering: Opt-out markers.

Search and duplicates

Once a project is indexed, semanticastindexer (SAI) gives you three distinct retrieval capabilities over the stored vectors. Each answers a different question, and each is available both as a CLI subcommand and as a shipped MCP tool, backed by one shared core (src/search.rs) so the CLI and the MCP server always agree:

All three open the index read-only, so a search can run while an index is open elsewhere. The similar and duplicates subcommands need a full build (--features all is recommended) so every backend and embedder is available. The top-level --backend / --embedder / --collection / --config flags still apply (before or after the subcommand) and pick up the YAML defaults.

The scores below are cosine similarity (higher = more alike). What counts as “similar enough” depends on your embedding model — see Tuning similarity before you trust a threshold.

Semantic search — “where is X?”

Use semantic search to find code by meaning rather than by literal text. The query is embedded as a query vector and matched against the nearest indexed chunks. This is the right tool for exploratory questions (“where do we create the Qdrant collection?”, “how do we parse transcripts?”) where you do not yet have a code sample in hand.

# Search only (read-only — does not upload the codebase).
./target/release/semanticastindexer --query-only --collection source_code \
    --query "where do we create the qdrant collection"

# --query without --query-only indexes first, then searches.
./target/release/semanticastindexer --root src --ext ts,tsx \
    --query "retry with exponential backoff"

--limit controls how many results are printed (default 5). Each result line shows the matched chunk’s score, path, line range, and symbol.

The MCP equivalent is sai_search_code, which embeds the query and returns the nearest indexed chunks. It additionally supports post-filters that the bare --query CLI path does not expose:

{
  "query": "retry with exponential backoff",
  "limit": 8,
  "language": "ts",
  "path_glob": "src/**",
  "include_text": false
}

• query (required) — natural-language or code query.
• limit — max results, clamped to 50 (default 8).
• language — keep only hits whose stored language label matches (e.g. "ts").
• path_glob — keep only hits whose path matches the glob (e.g. "src/**").
• include_text — return the full chunk text instead of a capped snippet (default false).

Find similar — “what looks like this?”

Use find-similar when you already have a piece of code and want its nearest neighbours. There are two distinct modes, and you provide exactly one of them.

By snippet (--code)

The snippet is embedded as a passage (code-vs-code space) and used to search for neighbours. Use this for code you have not indexed yet — a snippet from a PR, a function you are about to add, or text pasted from elsewhere.

./target/release/semanticastindexer similar \
    --code "function formatDuration(s) { return s }" --limit 8

similar --code needs a local embedder (the DuckDB backend). Qdrant embeds server-side, so --code against --backend qdrant returns a clear error. See Backends and embedders.

By existing chunk (--path + --line)

This locates an already-indexed chunk by its path and 1-based start line, reuses its stored vector (no re-embedding), and searches for neighbours with the chunk itself excluded from its own results. Use this to ask “what else in the codebase resembles this specific function I am looking at?”.

./target/release/semanticastindexer similar \
    --path src/utils/transcriptParser.ts --line 103 --min-score 0.0

If there is no indexed chunk at that exact path:line, you get a clear no indexed chunk at <path>:<line> error.

Reading the output and threshold

similar prints one line per neighbour, ranked by score descending:

score  path:start-end  symbol

Provide exactly one of --code or --path and --line — anything else is a clear error. --min-score resolves flag > config find_similar_min_score (0.85) > default. Pass --min-score 0 to see the raw score distribution before picking a cut.

The MCP tool sai_find_similar takes the same two modes — provide either code or both path and line; anything else is a parameter error:

{ "code": "function formatDuration(s) { return s }", "limit": 8 }

{ "path": "src/utils/transcriptParser.ts", "line": 103, "min_score": 0.0 }

• code — snippet to embed as a passage (mutually exclusive with path/line).
• path + line — locate an existing chunk by path and 1-based start line.
• limit — max results, clamped to 50 (default 8).
• min_score — drop results below this cosine cut. When omitted, sai_find_similar falls back to the configured find_similar_min_score, so omitting the arg still applies the model-tuned cut; pass an explicit 0.0 to see the raw distribution.

Find duplicates — “where do we repeat ourselves?”

Use find-duplicates for a codebase-wide near-duplicate audit. Unlike the other two tools you give it no query: it scans every stored chunk and reports clusters of chunks that are near-identical to one another. It works on stored vectors only (no re-embedding), so it runs on either backend.

How it works: for each chunk it takes that chunk’s top_k nearest neighbours, keeps each edge whose similarity is >= min_score, and unions the connected chunks into clusters (union-find). Clusters with at least min_cluster_size members are returned largest-first (tie-break: higher max edge similarity), truncated to max_clusters.

# Use config / built-in thresholds.
./target/release/semanticastindexer duplicates

# Tune the knobs and restrict the scan to a subtree.
./target/release/semanticastindexer duplicates \
    --min-score 0.85 --top-k 10 \
    --min-cluster-size 2 --max-clusters 20 \
    --path-glob "src/utils/**"

Each knob resolves CLI flag > config (similarity.*) > built-in default:

--path-glob restricts which chunks are scanned.

Reading the output

Clusters print largest-first, each with its size and the min/max edge similarity inside the cluster:

cluster (size N, sim min..max):
  path:start-end  symbol
  ...

A higher min..max band means the members are tighter copies of each other; a lower band means a looser family. If you are getting too many or too few clusters, raise or lower --min-score — see Tuning similarity.

The MCP tool sai_find_duplicates exposes the same algorithm with the same knobs (each resolving tool arg > config value > built-in default):

{
  "min_score": 0.85,
  "min_cluster_size": 2,
  "top_k": 10,
  "max_clusters": 20,
  "path_glob": "src/utils/**"
}

• min_score — minimum cosine similarity for an edge to count.
• min_cluster_size — smallest cluster to report.
• top_k — nearest-neighbour fan-out per chunk (clamped to 50).
• max_clusters — max clusters returned, largest first (default 50).
• path_glob — restrict the scan to matching paths.

Opting chunks out

Chunks marked with the opt-out marker are excluded from duplicate clustering entirely — both as a cluster seed and as a neighbour of other chunks — so a deliberately-repeated helper never pollutes the report. See Opt-out markers.

Choosing the right tool

• You have a question, not code → semantic search (sai_search_code / --query).
• You have a snippet or one specific chunk and want its neighbours → find-similar (sai_find_similar / similar).
• You want a repository-wide repetition audit → find-duplicates (sai_find_duplicates / duplicates).

Using these tools from an agent

For the agent-facing workflow — when to reach for each tool while coding, and a triage protocol that judges every duplicate/similarity finding (read the real source → classify real / boilerplate / intentional / fragment → propose a verified fix) before acting — see the sai-deslop skill (.agents/skills/sai-deslop/). For a repo-wide audit in Claude Code, delegate to the dedup-auditor subagent (.claude/agents/), which runs the sweep in an isolated context and returns a classified digest. Both are summarized under MCP server → Skills & subagents.

See also

• Tuning similarity — picking model-appropriate thresholds.
• Output schemas — exact JSON shapes returned by the MCP tools.
• MCP server — full tool catalog and wiring.

Keeping the index in sync

A full re-index is the right move once. After that, you want the vector store to track your commits without re-walking the whole tree. That is what sai sync is for: it re-indexes only the files that changed, so the index stays in lockstep with HEAD. Wire it into git hooks and you never have to think about it again.

This page is a hook recipe set. For the underlying flags see the CLI reference; for first-time and full re-indexing see the indexing guide.

How sync decides what changed

sync computes a changed-file set, then for each path it does the same thing: delete that file’s existing points, then upload the current on-disk content fresh. There are three ways to supply the changed set, in priority order:

--file wins outright: when one or more --file paths are given, git is not consulted at all. Otherwise --staged runs git diff --cached, and the default path runs git diff against --since, whose default is HEAD~1 — i.e. the diff HEAD~1..HEAD, the files in the most recent commit.

If the changed set is empty, sync prints sync: no changed files and exits cleanly. That makes it safe to fire from a hook on every commit, even commits that touch nothing indexable.

Each changed file: deleted, then re-uploaded

For every path in the changed set, sync first deletes that file’s existing points — always, unconditionally — and then decides whether to re-upload:

• File exists and is indexable (wanted extension, passes the sai-cfg.yml globs, not an autogenerated file, and produces at least one chunk): it is chunked, embedded, and upserted fresh. Reported as + <path> (N chunks).
• File is gone, excluded, or empty of indexable content: the delete stands and nothing is re-uploaded. Reported as - <path> (<reason>).

The removal reasons you will see come straight from the binary:

This is the key property: a changed file that has become a test file, a generated file, or has been deleted is dropped from the collection rather than re-added. sync honors the exact same extension, glob, and generated-marker rules as a full index — so the index never drifts to include files a full re-index would skip. (See the opt-out markers guide for the autogenerated marker, and configuration for the exclusion config.)

At the end you get a one-line summary:

sync: 3 file(s) re-indexed (47 chunks), 1 file(s) removed

Hook templates

All hooks below assume the binary is at an absolute path and that QDRANT_URL / QDRANT_API_KEY are set in the hook’s environment when you use the Qdrant backend (offline DuckDB needs no such env). Use --silent in hooks to suppress timing, progress, and dirty-state warnings — the noise-free mode built for hooks and CI.

Run hooks from the repo root. sync resolves git diff paths against the current directory and stores them as each chunk’s payload path. Git runs hooks from the repo root by default, so the paths git reports match the stored payload paths — keep it that way. If you indexed a subtree with --root, keep your hook’s working directory and flags consistent with how you first indexed.

pre-commit — sync the staged set

.git/hooks/pre-commit re-indexes exactly what is about to be committed:

#!/bin/sh
# Re-index staged changes before the commit lands.
SAI=/abs/path/to/semanticastindexer
QDRANT_URL="https://<id>.<region>.aws.cloud.qdrant.io:6334" \
QDRANT_API_KEY="$QDRANT_API_KEY" \
  "$SAI" --ext ts,tsx sync --staged --silent >/dev/null 2>&1 &
exit 0

The trailing & backgrounds the sync and exit 0 lets the commit proceed immediately (see Don’t block the commit below).

post-commit / post-merge — sync the new commit

.git/hooks/post-commit (and the identical .git/hooks/post-merge) keep the index in lockstep with the commit that just landed:

#!/bin/sh
# Re-index the files in the commit/merge that just completed.
SAI=/abs/path/to/semanticastindexer
QDRANT_URL="https://<id>.<region>.aws.cloud.qdrant.io:6334" \
QDRANT_API_KEY="$QDRANT_API_KEY" \
  "$SAI" --ext ts,tsx sync --since HEAD~1 --silent >/dev/null 2>&1 &

--since HEAD~1 diffs the last commit, which is exactly the set of files just committed or merged. sync respects the same --ext and sai-cfg.yml filters as a full index.

pre-push — sync everything since the upstream

.git/hooks/pre-push is a good catch-all if you commit without per-commit hooks and want the index correct before code leaves your machine. Diff against the upstream tracking branch:

#!/bin/sh
# Re-index everything not yet on the upstream branch, before pushing.
SAI=/abs/path/to/semanticastindexer
UPSTREAM=$(git rev-parse --abbrev-ref --symbolic-full-name '@{u}' 2>/dev/null)
if [ -n "$UPSTREAM" ]; then
  "$SAI" --ext ts,tsx sync --since "$UPSTREAM" --silent >/dev/null 2>&1 &
fi
exit 0

--since <rev> accepts any revision git understands, so @{u} (the upstream of the current branch) gives you the full set of files in unpushed commits.

husky

With husky, put the same commands in the managed hook files. For example .husky/post-commit:

SAI=/abs/path/to/semanticastindexer
"$SAI" --ext ts,tsx sync --since HEAD~1 --silent >/dev/null 2>&1 &

And .husky/pre-commit:

SAI=/abs/path/to/semanticastindexer
"$SAI" --ext ts,tsx sync --staged --silent >/dev/null 2>&1 &
exit 0

lefthook

With lefthook, add the syncs to lefthook.yml. Background the command so lefthook continues without waiting:

pre-commit:
  commands:
    sai-sync:
      run: /abs/path/to/semanticastindexer --ext ts,tsx sync --staged --silent >/dev/null 2>&1 &

post-commit:
  commands:
    sai-sync:
      run: /abs/path/to/semanticastindexer --ext ts,tsx sync --since HEAD~1 --silent >/dev/null 2>&1 &

Don’t block the commit

Embedding takes time, and the Qdrant backend talks to the network. Two pieces keep a git commit from stalling on it:

• & — background the sync so the hook returns immediately and git proceeds.
• >/dev/null 2>&1 — discard stdout and stderr so the backgrounded process does not bleed output into your terminal after the prompt returns. Pair it with --silent to keep the process quiet in the first place.

Add exit 0 to hooks where the background process is the last statement, so the hook’s exit status does not depend on the still-running sync.

Trade-off. Backgrounding means the commit succeeds even if the sync later fails (bad credentials, backend down). The index will be momentarily behind, and the next sync (or a full re-index) repairs it. If you would rather a broken index fail the commit, drop the & and let sync run in the foreground — and accept the latency.

A note on cost (DuckDB HNSW)

sync always wraps its work in a bulk window (begin_bulk / end_bulk). On Qdrant this is a no-op. On the DuckDB backend it is required for correctness, and it is not free: every sync drops the experimental HNSW index up front and rebuilds it at the end. DuckDB’s HNSW returns too few candidates after in-place deletes — recall degrades — and hnsw_compact_index does not fix it; dropping and rebuilding the index is the only way to restore full recall.

The practical consequence: syncing a single file on DuckDB still pays a full index rebuild. For a small or medium index this is cheap; for a very large one it can dominate the hook’s runtime, which is another reason to background the call. See performance for the rebuild cost, and backends and embedders for the DuckDB vs. Qdrant trade-offs.

See also

• CLI reference — every sync flag and default.
• Indexing — first-time and full re-indexing.
• CI/CD — keeping the index fresh from pipelines instead of local hooks.
• Opt-out markers — how files are excluded and the autogenerated marker.
• Performance — the DuckDB HNSW rebuild cost.

CI/CD integration

SAI is a single static binary that runs the same in CI as it does on your laptop: walk a tree, embed chunks, push them to a vector backend, and report duplicates. This guide shows how to wire it into a pipeline — keeping the index in sync on every push, caching the local ONNX model, running Ollama as a service, passing Qdrant credentials as secrets, and failing the build when new near-duplicates appear.

The repo dogfoods one indexing workflow. dedup-gate.yml builds the PR’s own binary, fully indexes the base branch into a per-PR Qdrant collection, runs the real sync --since to bring the index to the PR head, and fails the PR only when the near-duplicate cluster count in src/ grows (see Fail the build when new duplicates appear). release.yml (release binaries) and docker.yml (the container image described below) cover packaging. The indexing examples below remain templates for your repository.

Run SAI from the prebuilt container

Prebuilt images are published to the GitHub Container Registry, so a job can run SAI without compiling it:

• ghcr.io/maadgrom/semanticastindexer:latest — Alpine, lean (musl): qdrant + duckdb + ollama + ast + mcp, no local ONNX embedder. Small; for the Qdrant (server-side inference) and Ollama CI paths.
• ghcr.io/maadgrom/semanticastindexer:latest-full — glibc, --features all: adds the ort on-device ONNX embedder. The model + tokenizer download on first use; persist them across runs by mounting a volume (or restoring a CI cache) at HF_HOME, e.g. docker run -e HF_HOME=/hf-cache -v hf-cache:/hf-cache ....

Use it as the job container:

jobs:
  dedup-gate:
    runs-on: ubuntu-latest
    container: ghcr.io/maadgrom/semanticastindexer:latest
    steps:
      - uses: actions/checkout@v4
      - name: Index against Qdrant (server-side inference, no local model)
        run: semanticastindexer --root src --ext ts,tsx --backend qdrant --silent
        env:
          QDRANT_URL: ${{ secrets.QDRANT_URL }}
          QDRANT_API_KEY: ${{ secrets.QDRANT_API_KEY }}
      - name: Fail if new near-duplicates appear
        run: semanticastindexer duplicates --backend qdrant --min-score 0.88

Or one-shot with docker run (mount the repo, pass the key as an env var):

docker run --rm -v "$PWD:/repo" -w /repo \
  -e QDRANT_URL -e QDRANT_API_KEY \
  ghcr.io/maadgrom/semanticastindexer:latest duplicates --backend qdrant --min-score 0.88

Every image bundles git, so sync --since / --staged work inside it. Tags: :X.Y.Z and :latest (releases), :edge (main), :sha-<short>, each with a -full companion.

CI is non-interactive by design

Every yes/no prompt in SAI auto-declines when there is no terminal attached. stdin is not a TTY in CI, so each prompt returns “No” immediately and the run continues — it never blocks waiting for input and never takes a destructive action by default.

Two prompts behave this way:

• Dimension-mismatch on the DuckDB index — if an existing local index was built with a different embedding model, an interactive run offers to delete and rebuild it (defaulting to No). In CI the prompt is skipped and the underlying error surfaces instead, so a stale index can never be silently wiped.
• Dirty-tree warning on duplicates — when the index contains chunks stamped from an uncommitted working tree, an interactive run asks whether to proceed. In CI the warning is printed to stderr and the command proceeds.

Add --silent to suppress timing, progress, and dirty warnings entirely — it is built for hooks and CI and keeps logs clean:

semanticastindexer sync --silent

Passing Qdrant credentials as secrets

When you target the Qdrant backend, the API key is read only from the environment (it is a secret); the cluster URL can come from qdrant.url in sai-cfg.yml or the QDRANT_URL env var. In CI the simplest is to pass both as secrets and never commit the key:

In GitHub Actions, store both as repository secrets and expose them through env:

env:
  QDRANT_URL: ${{ secrets.QDRANT_URL }}
  QDRANT_API_KEY: ${{ secrets.QDRANT_API_KEY }}

If a key is ever exposed, rotate it in the cluster’s API Keys tab. See Configuration → Environment variables for how the URL and key are resolved.

Sync on every push (Qdrant backend)

The sync subcommand re-indexes only the files that changed in a revision range, so it is cheap to run on every push. By default it diffs HEAD~1..HEAD; pass --since for a different base. This workflow keeps a Qdrant collection current:

name: Index code
on:
  push:
    branches: [main]

jobs:
  index:
    runs-on: ubuntu-latest
    env:
      QDRANT_URL: ${{ secrets.QDRANT_URL }}
      QDRANT_API_KEY: ${{ secrets.QDRANT_API_KEY }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2 # need HEAD~1 for the default --since

      - name: Install SAI
        run: curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash

      - name: Sync changed files
        run: semanticastindexer --backend qdrant --ext ts,tsx sync --since HEAD~1 --silent

sync deletes each changed file’s old points and uploads the current content fresh; files that are gone (deleted or now excluded) are removed from the collection. For the full mechanics — staged diffs (--staged), explicit --file lists, and how it pairs with git hooks — see keeping in sync.

Caching the Hugging Face ONNX model (ort embedder)

The default ort embedder runs ONNX Runtime on-device with no server and no API keys. On the first run it pulls the model from Hugging Face — the code-trained jinaai/jina-embeddings-v2-base-code (or intfloat/multilingual-e5-small for the text default). That download repeats on every fresh runner unless you cache it.

The model is fetched into the Hugging Face hub cache (~/.cache/huggingface), so cache that directory between runs:

jobs:
  index:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install SAI
        run: curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash

      - name: Cache Hugging Face model
        uses: actions/cache@v4
        with:
          path: ~/.cache/huggingface
          # bump the key when you change `model` so a new model is re-downloaded
          key: hf-jina-code-v2

      - name: Index with the local ort embedder
        run: semanticastindexer --backend duckdb --embedder ort --root src --ext ts,tsx --silent

This keeps the DuckDB index entirely local to the runner — no Qdrant credentials needed. If you also persist the local index file across runs, the dimension-mismatch prompt becomes relevant: it auto-declines in CI, so a model change surfaces an error rather than silently rebuilding (see above).

Running Ollama as a service in CI

The ollama embedder talks to an embedding server over HTTP, which suits CI where an embedding service often already runs. Start ollama serve, pull an embedding model, then point SAI at it. Configure ollama.url and ollama.model in sai-cfg.yml (see reference/configuration.md):

jobs:
  index:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install SAI
        run: curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash

      - name: Start Ollama and pull an embedding model
        run: |
          curl -fsSL https://ollama.com/install.sh | sh
          ollama serve &
          # wait for the server, then pull the model
          until curl -sf http://localhost:11434/api/tags >/dev/null; do sleep 1; done
          ollama pull mxbai-embed-large

      - name: Index via the Ollama embedder
        run: semanticastindexer --backend duckdb --embedder ollama --root src --ext ts,tsx --silent

See the Ollama integration guide for matching the model to the configured vector_dim.

Fail the build when new duplicates appear

The duplicates subcommand scans stored vectors for near-duplicate clusters and prints them human-readably. When nothing crosses the threshold it prints a line starting with no near-duplicate clusters; when something does it prints N near-duplicate cluster(s): followed by each cluster. You can gate a PR on that output.

The simplest gate fails when any cluster is found above your chosen threshold:

#!/usr/bin/env bash
set -euo pipefail

# Index the working tree first (local ort backend, no creds needed).
semanticastindexer --backend duckdb --embedder ort --root src --ext ts,tsx --silent

out=$(semanticastindexer duplicates --backend duckdb --min-score 0.95)
echo "$out"

if echo "$out" | grep -q '^no near-duplicate clusters'; then
  echo "No duplicates above threshold."
else
  echo "::error::near-duplicate clusters detected"
  exit 1
fi

The threshold knobs map directly to flags: --min-score (minimum cosine similarity for an edge), --min-cluster-size, --top-k, and --path-glob to scope the scan. Each resolves CLI flag > config similarity.* > built-in default. Tune them with tuning similarity and the search and duplicates guide.

To catch only newly introduced duplicates rather than failing on a pre-existing backlog, scope the scan to changed files with --path-glob, or compare the cluster count of the base branch against the head branch and fail only when it grows. This repository’s own dedup-gate.yml implements the base-vs-head count comparison.

Read-only by default

The CLI read commands — duplicates, similar, and --query-only — open the index read-only, and the MCP server is read-only unless started with --allow-write. A CI job that only searches or reports duplicates can never mutate the collection. See the CLI reference for the full flag set.

See also

• Keeping in sync — sync, git hooks, and revision ranges.
• Environment reference — QDRANT_URL, QDRANT_API_KEY, and friends.
• Troubleshooting — what to do when a CI run errors out.

Choosing a model

The quality of your search and near-duplicate results depends almost entirely on the embedding model you pick. This guide helps you choose an embedder (the thing that turns code into vectors) and a model that suits your task — and shows the YAML you need so the pieces line up.

The two settings that matter most are embedder (how vectors are produced) and model (which weights). A third, vector_dim, must match the model — get it wrong and SAI errors at runtime.

Two embedders for the DuckDB backend

The DuckDB backend produces vectors locally through a pluggable embedder, selected with embedder: ort | ollama in sai-cfg.yml (or --embedder <name> on the CLI). The default is ort.

Pick ort when you want a self-contained, offline-after-first-run binary. Pick ollama when you already run Ollama and want to reuse its model library (or want a larger, higher-dimension model than the bundled ONNX defaults).

The third backend, qdrant, defaults to embedder: qdrant — Qdrant Cloud server-side inference (no local embedder). But with embedder: ort (or ollama) it embeds on-device and upserts raw vectors — so a code model like jinaai/jina-embeddings-v2-base-code (768-d) is usable against self-hosted / OSS Qdrant, not just the Cloud path. See Backends & embedders and Qdrant Cloud.

Models at a glance

Why a code-trained model matters for de-dup

intfloat/multilingual-e5-small is a multilingual text model. Distinct functions written in the same language all embed at roughly 0.91 cosine similarity to each other, so duplicates collapses everything into one giant mega-cluster — it cannot tell real near-duplicates from merely “both are Rust.”

A code-trained embedder such as jinaai/jina-embeddings-v2-base-code spreads unrelated functions far apart and surfaces genuine near-duplicates (even when the two copies have different names). That is why it is the default for the offline ort path. Code models also run at lower absolute cosine scores than e5, so their duplicate thresholds sit lower (see below and Tuning similarity).

vector_dim MUST match the model

vector_dim is validated at runtime. A mismatch is a hard error, e.g.:

embedder produced 768-d vectors but vector_dim=384 …

Use these values:

• jinaai/jina-embeddings-v2-base-code → 768
• intfloat/multilingual-e5-small → 384
• nomic-embed-text → 768
• mxbai-embed-large → 1024

If you change the model (and therefore vector_dim) on an existing DuckDB index, you need a fresh index: delete .index/code.duckdb or run with --recreate.

Recipes

Default (offline ONNX, code-trained) — recommended

This is what you get with the ort embedder and no model set, stated explicitly:

backend: duckdb
embedder: ort
model: jinaai/jina-embeddings-v2-base-code   # code-trained (CodeSearchNet)
vector_dim: 768                              # MUST match the model
prefix_style: none                           # symmetric model — no passage:/query: prefix
duckdb:
  model_repo: jinaai/jina-embeddings-v2-base-code   # ort downloads onnx/model.onnx + tokenizer.json
similarity:
  duplicate_min_score: 0.88                  # code models run lower than e5

First-run note for this repo. The pinned hf-hub cannot fetch tokenizer.json from the Jina repo (Hugging Face Xet storage). Stage it once into the HF cache before the first index — see the caveat in Backends & embedders. The onnx/model.onnx download itself works.

Ollama (HTTP server)

The ollama embedder requires ollama.model — there is no E5 fallback, and construction fails clearly if it is unset. Start the server and pull the model first:

ollama serve
ollama pull nomic-embed-text

backend: duckdb
embedder: ollama
model: nomic-embed-text   # informational label
vector_dim: 768           # nomic-embed-text is 768-d
ollama:
  url: http://localhost:11434   # default
  model: nomic-embed-text       # required — the model Ollama actually runs

For mxbai-embed-large instead, set ollama.model: mxbai-embed-large and vector_dim: 1024.

Ollama text models are not code-trained, so expect the same de-dup limitation as e5: index and search work, but you will likely need to retune the duplicate thresholds. See Tuning similarity.

Prefix styles auto-detect

Embedding prefixes (the passage:/query: scheme E5 was trained with) are chosen automatically from the model name when you do not set prefix_style:

• model name contains e5 → e5 (asymmetric passage: / query:)
• model name contains qwen → qwen (bare passages, instructed query)
• otherwise → none (both sides bare)

So the Jina code model auto-detects to none (correct — it is symmetric), and e5 auto-detects to e5. Override with prefix_style: e5 | qwen | none only when the auto-detection guesses wrong for your model — for example a non-E5 Ollama model that should run with no prefix. Relevance can suffer if you force an asymmetric prefix on a model that was not trained with one.

Offline / cached models (ort)

The ort embedder downloads from duckdb.model_repo on first run and caches the result. To run fully offline (air-gapped CI, no network), point duckdb.model_cache at a pre-populated Hugging Face cache directory:

duckdb:
  model_repo: jinaai/jina-embeddings-v2-base-code
  model_cache: /path/to/huggingface/cache   # reuse a pre-populated HF cache, no network

Where to find models

• ONNX models for ort: any Hugging Face repo that ships onnx/model.onnx and tokenizer.json. Browse Hugging Face ONNX models. The defaults live at jinaai/jina-embeddings-v2-base-code and Xenova/multilingual-e5-small.
• Embedding models for ollama: browse Ollama embedding models, including mxbai-embed-large and nomic-embed-text.

See also

• Backends & embedders — the full backend/embedder reference and the Jina first-run staging caveat.
• Configuration — every sai-cfg.yml key, including the duckdb and ollama sub-sections.
• Tuning similarity — adjusting duplicate_min_score and the other thresholds once you pick a model.

Tuning similarity

Semantic search ranks results by cosine similarity: a number between roughly 0.0 and 1.0 where higher means “closer in embedding space”. Whether a given score means “a real near-duplicate” or “loosely related” depends entirely on the embedding model you chose. This page explains the four threshold knobs SAI exposes, how they resolve, and a concrete workflow for picking values that fit your model.

For what these thresholds gate (the similar / duplicates features themselves), see Search and duplicates. For where to put the config, see Configuration. For the MCP tool args, see MCP server.

The four knobs

All four live in the similarity: block of sai-cfg.yml:

find_duplicates works by taking each stored chunk, fetching its top_k nearest neighbours, keeping every edge whose similarity is >= duplicate_min_score, and running union-find over the kept edges. Components with at least duplicate_min_cluster_size members become reported clusters (largest first).

find_similar_min_score is a flat post-filter on a single query’s results. duplicate_min_score is an edge threshold inside the clustering graph, which is why its default sits higher: a duplicate should be a tighter match than a merely “similar” result.

Resolution order

Every knob resolves the same way, per knob, independently:

CLI flag / MCP tool arg  >  config (similarity.*)  >  built-in default

So if sai-cfg.yml sets duplicate_min_score: 0.90 but a sai_find_duplicates call passes min_score: 0.95, the call uses 0.95. Omit the arg and the call falls back to the configured 0.90; omit the config key too and it falls back to the built-in 0.93.

# sai-cfg.yml
similarity:
  find_similar_min_score: 0.80
  duplicate_min_score: 0.90
  duplicate_min_cluster_size: 2
  top_k: 12

Every field is optional — a partial similarity: block (or none at all) just uses built-in defaults for whatever you leave out.

Cutoffs are model-specific

Tune the thresholds per model. A cosine of 0.85 does not mean the same thing across embedders: a code-trained model (e.g. Jina code, the default for the ort embedder) generally produces lower raw cosines for the same pair of snippets than a general text model like multilingual-e5-small, and a Qwen-style model differs again. Concretely, a Qwen3 cosine of 0.85 is a looser match than e5’s 0.85, so the value that cleanly separates “duplicate” from “merely similar” is different for each.

That means the built-in defaults (0.85 / 0.93) are a starting point, not a universal truth. If you switch the model — or the embedder, which changes the default model — re-tune the similarity: block. See Choosing a model for the model/embedder pairings.

Workflow: read the distribution, then set thresholds

Don’t guess. Turn the filter off, look at the actual scores your model produces on your code, then pick cutoffs from the gap in the distribution.

1. See the raw scores. Call find_similar with min_score set to 0 so nothing is filtered out:

# CLI: neighbours of an existing indexed chunk, unfiltered
semanticastindexer similar \
  --path src/utils/parse.ts --line 42 \
  --limit 20 \
  --min-score 0

# CLI: neighbours of an inline snippet, unfiltered
semanticastindexer similar \
  --code 'function clamp(x, lo, hi) { return Math.max(lo, Math.min(hi, x)); }' \
  --limit 20 \
  --min-score 0

From an MCP client, call sai_find_similar with min_score: 0:

{ "name": "sai_find_similar",
  "arguments": { "path": "src/utils/parse.ts", "line": 42, "limit": 20, "min_score": 0 } }

2. Read the gap. You’ll typically see a cluster of high scores (the genuine matches) then a drop-off into noise. Pick find_similar_min_score just above the noise floor and duplicate_min_score up near the tight top of the distribution.

3. Probe the duplicate edge threshold. Run duplicates with a deliberately low --min-score to see what clusters at all, then raise it until only true near-duplicates survive:

# Start permissive to see everything that clusters, then tighten min_score upward
semanticastindexer duplicates --min-score 0.80 --min-cluster-size 2 --top-k 10

4. Write the chosen values into similarity: in sai-cfg.yml so every search, CLI run, and MCP server picks them up by default. The CLI flags / MCP args remain available for one-off overrides.

Scoping duplicates with a path glob

Both the duplicates CLI subcommand and the sai_find_duplicates tool accept a path glob to restrict the scan to part of the tree — useful for hunting copy-paste within one module without scanning the whole repo:

# Only cluster chunks under src/utils
semanticastindexer duplicates --path-glob 'src/utils/**' --min-score 0.93

{ "name": "sai_find_duplicates",
  "arguments": { "path_glob": "src/utils/**", "min_score": 0.93, "min_cluster_size": 2 } }

The glob filters which chunks enter the scan; the threshold knobs above still decide which of those chunks cluster together.

Knob-tuning cheatsheet

• Too many weak “similar” hits → raise find_similar_min_score.
• Real matches getting dropped → lower find_similar_min_score (re-check the raw distribution first).
• Duplicate clusters that aren’t actually duplicates → raise duplicate_min_score.
• Known copies not clustering → lower duplicate_min_score, or raise top_k so the neighbour fan-out reaches them.
• Want only larger duplicate groups → raise duplicate_min_cluster_size (it defaults to 2, the smallest meaningful cluster).

See also

• Search and duplicates — running the searches these knobs tune.
• Configuration — the full sai-cfg.yml schema.
• MCP server — sai_find_similar / sai_find_duplicates tool args.
• Choosing a model — why the model changes what a score means.

Opt-out markers

Opt-out markers are small comments you place directly in your source to control how SAI treats a piece of code. They give you per-function (or per-window) control over two independent things: whether a chunk is indexed at all, and whether it participates in near-duplicate clustering.

There are exactly two markers:

Use sai-noindexing to keep code out of the index completely (vendored snippets, intentionally noisy boilerplate). Use sai-noduplicate when code should stay findable by search but you do not want it flagged as a duplicate — for example, deliberately parallel test fixtures or per-language mirrors of the same routine.

How detection works

Detection is a case-insensitive substring match on the raw source — the original file text, before comment stripping. Because it is a plain substring scan, it is language- and comment-syntax agnostic. All of these are detected:

// sai-noindexing

# sai-noduplicate

-- sai-noindexing

<!-- sai-noduplicate -->

The marker only has to appear somewhere inside the chunk’s line span; it does not need to be on its own line, and casing is ignored (sai-NoIndexing, SAI-NODUPLICATE, and sai-noindexing all match).

Granularity

What a single marker affects depends on the chunker (see chunking.md):

• AST chunker (chunker: ast, covering TypeScript/TSX, Rust, Go, and Python): each function is one chunk, so a marker applies to the function whose body contains it. Place the marker on the line above the function or anywhere inside it.
• Lines chunker (chunker: lines, the default and the fallback for every other language): chunks are line windows (up to ~60 lines, with 8-line overlap), so a marker applies at window granularity. A marker near a window boundary may only drop the window(s) whose line range contains it — the overlapping neighbour window can survive.

For predictable results, prefer the AST chunker when your code is TS/TSX/Rust/Go/Python, and put the marker right next to the function it should govern.

Examples

sai-noindexing — the function is never embedded or stored:

// sai-noindexing
function internalHelper() {
  // This function is never indexed, so it can't be searched or matched.
}

sai-noduplicate — the function stays indexed and searchable, but is excluded from duplicate clustering:

// sai-noduplicate
function intentionallySimilar() {
  // Findable by search; never grouped into a duplicate cluster.
}

Disabling marker handling

Both markers are honored by default. Two config toggles let you turn them off independently (both default to true):

honor_noindex_marker: true        # respect sai-noindexing comments
honor_noduplicate_marker: true    # respect sai-noduplicate comments

Set honor_noindex_marker: false to index even chunks containing sai-noindexing, and honor_noduplicate_marker: false to include sai-noduplicate chunks in duplicate clustering. When a toggle is off, that marker is ignored — the other toggle is unaffected. See configuration.md for where these keys live in sai-cfg.yml.

Caveat: string literals also trigger the markers

Because detection is a raw substring match (it runs before comments are stripped, and it does not parse the code), the literal marker text inside a string literal triggers the opt-out just like a comment would. For example:

function logUsage() {
  // Oops: this string literal contains the marker text verbatim.
  console.log("the sai-noindexing flag was set");
}

The function above would be skipped from indexing even though no comment marks it. Avoid embedding the literal strings sai-noindexing or sai-noduplicate in code unless you actually intend the opt-out.

Related pages

• Chunking reference — how functions and line windows become chunks.
• Configuration reference — the honor_noindex_marker / honor_noduplicate_marker keys.
• Search and duplicates — what duplicate clustering does and which tools it powers.

CLI reference

The semanticastindexer (SAI) binary is a single CLI with an optional subcommand. With no subcommand it runs a full index of --root; subcommands cover storage maintenance (flush), incremental re-indexing (sync), similarity search (similar, duplicates), and the MCP server (mcp).

Run the binary from the target project’s repo root so stored payload paths stay project-relative (or point --root at the project’s source dir). Connection settings for the Qdrant backend are read from the environment, never hard-coded — see environment variables (QDRANT_URL, QDRANT_API_KEY). Persistent configuration lives in sai-cfg.yml; see the configuration reference for every key.

Synopsis

semanticastindexer [GLOBAL FLAGS] [INDEX FLAGS]            # default: full index of --root
semanticastindexer [GLOBAL FLAGS] init       [INIT FLAGS]
semanticastindexer [GLOBAL FLAGS] flush
semanticastindexer [GLOBAL FLAGS] sync       [SYNC FLAGS]
semanticastindexer [GLOBAL FLAGS] similar    [SIMILAR FLAGS]
semanticastindexer [GLOBAL FLAGS] duplicates [DUPLICATES FLAGS]
semanticastindexer [GLOBAL FLAGS] mcp        [MCP FLAGS]

Resolution order

Every value resolves in this order:

CLI flag > sai-cfg.yml config value > built-in default.

A CLI flag always overrides the config file, which always overrides the compiled-in default. This applies to the global flags, the index-only flags, and the per-subcommand similarity thresholds documented below.

Global flags (before or after the subcommand)

These flags are declared global = true, so they are accepted in any position — before or after the subcommand name.

Note: --backend, --embedder, --config, --collection, -v/--verbose, --log-format, --timing, and --silent are global. The index-only flags below (--root, --ext, --chunker, --model, --query, --query-only, --recreate, --dry-run, --limit) are not global — they belong to the default (no-subcommand) index action.

Default action — full index (no subcommand)

With no subcommand, SAI walks --root, chunks each matching file, embeds the chunks, and upserts them into the collection. The flags below are specific to this default action.

Examples:

BIN="$(pwd)/target/release/semanticastindexer"
cd /path/to/your/project

# See exactly what would be indexed/skipped — no network, no upload.
"$BIN" --root src --dry-run

# Index the TypeScript tree into a named collection.
"$BIN" --root src --ext ts,tsx --collection source_code

# Index Go later, into the same collection.
"$BIN" --root path/to/go --ext go --collection source_code

# Search only (read-only — does not upload the codebase).
"$BIN" --query-only --collection source_code \
    --query "where do we create the qdrant collection" --limit 10

# Drop & rebuild the collection before indexing.
"$BIN" --root src --ext ts,tsx --recreate

Two distinct --limit defaults

--limit exists in two places with different defaults — do not confuse them:

DuckDB dimension-mismatch prompt

On the indexing path (not --query-only), if opening the DuckDB backend fails because the existing index file was built with a different embedding model (a vector-dimension mismatch), SAI offers to delete the file and re-index from scratch. The prompt defaults to NO, and on a non-interactive stdin (CI, git hooks, the MCP stdio server) it auto-declines immediately and propagates the original error — it is never destructive in automation. Declining (or piping non-TTY input) leaves the index untouched. --query-only never re-indexes, so it surfaces the error without offering to delete anything.

init

Generate a starter sai-cfg.yml — the standard, fully-commented config. By default it runs a short interview (vector backend, embedder, collection, model, plus optional connection settings and extra excluded directories); every question has a default accepted by pressing Enter. The model’s vector_dim is filled in automatically for recognized models (Jina code, e5 family, mxbai-embed-large, nomic-embed-text) and asked for otherwise. The generated file is validated before it is written.

semanticastindexer init             # interactive interview
semanticastindexer init --yes       # accept every default (non-interactive)

On a closed/non-interactive stdin every question falls back to its default, so init is safe in scripts and CI even without --yes. init never loads an existing config, so it also works when the current one is broken.

flush

Delete the entire collection from the vector storage.

semanticastindexer flush

flush takes no subcommand-local flags (only the global flags apply). Useful for the one-time re-index when point-ID hashing changed, or to fully reset a collection.

sync

Re-index only changed files — intended for git hooks (post-commit, post-merge, pre-commit). For each changed file: delete its existing points, then upload the current content fresh. Files that were deleted or are now excluded are removed from the collection. sync honors the same --ext and sai-cfg.yml filters as a full index.

Examples:

# Diff HEAD~1..HEAD (e.g. post-commit / post-merge).
semanticastindexer sync --since HEAD~1

# Staged changes (e.g. pre-commit).
semanticastindexer sync --staged

# Explicit file list (overrides git).
semanticastindexer sync --file src/a.ts --file src/b.ts

If git detection (or the explicit list) yields no changed files, sync prints sync: no changed files and exits. Run hooks from the repo root so the paths git reports match the stored payload paths. See the keeping the index in sync guide for full hook examples.

similar

Print the nearest neighbours of either a code snippet or an existing indexed chunk, as score path:start-end symbol. Requires a vector backend feature (duckdb or qdrant).

Exactly-one-of validation

similar requires exactly one target:

• either --code
• or both --path and --line

Any other combination is a clear error:

Examples:

# Nearest neighbours of a snippet (embedded as a passage — code-vs-code).
semanticastindexer similar --code "function formatDuration(s) { return s }" --limit 8

# Nearest neighbours of an existing indexed chunk (stored vector, self-excluded).
semanticastindexer similar --path src/utils/transcriptParser.ts --line 103 --min-score 0.0

The similar --code path needs a local embedder (the DuckDB backend). Qdrant embeds server-side, so --code against --backend qdrant returns a clear error. similar --path/--line works on either backend (it reuses the stored vector, no re-embed).

duplicates

Codebase-wide near-duplicate clusters. For each chunk, takes its nearest neighbours, keeps edges with similarity >= min-score, and unions them into clusters (union-find). Prints clusters largest-first. Uses stored vectors only (no re-embed). Requires a vector backend feature (duckdb or qdrant).

Note the resolution asymmetry: --min-score, --min-cluster-size, and --top-k each resolve flag > config (similarity.*) > built-in default, while --max-clusters resolves flag > built-in default 50 with no config key.

Output format:

N near-duplicate cluster(s) (min_score 0.93, min_cluster_size 2, top_k 10):
cluster (size 3, sim 0.9412..0.9871):
  src/a.ts:10-40  formatDuration
  src/b.ts:5-35   formatTime
  ...

When no clusters meet the thresholds, SAI prints a single no near-duplicate clusters (...) line.

Examples:

# Config / built-in thresholds.
semanticastindexer duplicates

# Fully specified.
semanticastindexer duplicates --min-score 0.85 --top-k 10 \
    --min-cluster-size 2 --max-clusters 20 \
    --path-glob "src/utils/**"

Dirty-index prompt

Before scanning, duplicates checks whether the index contains dirty-stamped chunks (chunks recorded from an uncommitted working tree). If so:

• On an interactive TTY, it warns and asks to proceed; the prompt defaults to NO, and declining aborts the scan (the results may reflect uncommitted work).
• On a non-interactive stdin (CI, git hooks, MCP), it prints the warning to stderr and proceeds without prompting — it never blocks.
• With --silent, the dirty check is skipped entirely.

Like the dimension-mismatch prompt, this never triggers a destructive or blocking action in automation.

mcp

Run the MCP server (semantic code search for AI agents) over stdio. Requires the mcp feature. When --backend/--embedder are unset, the MCP server applies its own defaults of duckdb + ollama (the offline, no-quota path); explicit CLI flags and config values still take precedence via the normal merge. The shipped tool names are sai_-prefixed: sai_search_code, sai_find_similar, sai_find_duplicates, sai_index_status, sai_prepare_mcp_setup, and (gated) sai_refresh.

By default the MCP server is read-only: it never modifies the index or the filesystem. --allow-write gates the only write tool (sai_refresh); --allow-setup gates execution of sai_prepare_mcp_setup.

# Read-only MCP server (default duckdb + ollama).
semanticastindexer mcp

# Enable the sai_refresh write tool.
semanticastindexer mcp --allow-write

# Also allow sai_prepare_mcp_setup to run the setup script.
semanticastindexer mcp --allow-write --allow-setup

See the MCP server and tools reference for full tool details, and the glossary for terminology.

update

Self-update: download and install the latest GitHub release over the current binary by running the official release installer. Config-independent — works from any directory, needs no indexer.yaml, and is always compiled in (no feature gate).

semanticastindexer update

On macOS and Linux the installer replaces the binary in place; the new version takes effect on the next invocation (restart any running MCP servers to pick it up). On Windows a running executable cannot overwrite itself, so update prints the exact PowerShell one-liner to run instead.

Feature gating

SAI is built with Cargo features. Subcommands and chunkers are compiled in only when the matching feature is present:

The --features all build enables every backend, embedder, and chunker so any project can be indexed fully offline (ort + DuckDB) out of the box. Requesting --chunker ast (or auto-selecting it) without the ast feature fails early with a clear, actionable error before any work begins.

Logging

Diagnostics (status, progress, warnings, operation timing) are emitted to stderr at a configurable level and format. Stdout is reserved exclusively for JSON-RPC frames (MCP) and CLI data output (query hits, --json, sync report, --dry-run log).

Three knobs — level, format, and timing — each resolve in the same order: the RUST_LOG env var / CLI flag first, then the project config’s logging: block, then a built-in default.

Timing controls only the per-operation span-close durations (index/embed/query/sync). The single end-of-command summary line (done … in Ns) is a plain info event and prints regardless — see timing & output.

Examples:

# Default (info level, pretty format, no per-op timing).
semanticastindexer --dry-run

# Debug-level logs, JSON format.
RUST_LOG=semanticastindexer=debug semanticastindexer --log-format json --query "my query"

# Trace level (very verbose), with per-operation timing breakdowns.
semanticastindexer -vv --timing --query "my query"

# Suppress logs (error level only).
semanticastindexer --silent

Project-wide defaults can live in sai-cfg.yml instead of flags — see the logging: config block. The flags and RUST_LOG always override it.

For MCP clients: diagnostics go to stderr and are captured in the client’s log file. The JSON-RPC stream (stdout) remains clean, enabling write-mode (--allow-write) even with strict JSON-RPC clients (e.g. Grok). Because the server is launched with a fixed command, the logging: block is the simplest way to raise its verbosity without editing each client’s launch config.

Timing and output

Every top-level command is wrapped in a timing helper (run_timed). On completion it prints a single line to stderr:

done at <git-sha>[, dirty] in <seconds>s

((dry-run) is appended for a dry-run.) This timing/summary line — along with progress lines, dirty warnings, and the “no config” note — is suppressed by the global --silent flag, which is recommended for hooks and CI. Indexing progress (embedded N/M chunks, per-file lines) is also written to stderr; the actual results (indexed ..., query hits, similarity output) go to stdout.

See also

• Configuration reference — every sai-cfg.yml key.
• Environment variables — QDRANT_URL, QDRANT_API_KEY.
• Glossary — backend, embedder, chunker, and marker terms.

Configuration reference

SAI reads a single YAML file, sai-cfg.yml, to decide what gets chunked, embedded, and stored. When --config <path> is not passed, SAI looks in the current directory for sai-cfg.yml, then sai-cfg.yaml, then the legacy indexer.yaml — the first one found wins. Generate a fully-commented starter file with semanticastindexer init (interactive; --yes accepts every default). Every value here is resolved in src/config.rs (build_plan), and every key in this page maps to a real field — nothing else is read.

The resolution order for most knobs is:

CLI flag  >  sai-cfg.yml value  >  built-in default

The Qdrant API key is the exception — it is a secret read only from the QDRANT_API_KEY environment variable, never from YAML (see Environment variables below). The Qdrant URL is a normal setting: qdrant.url in YAML, overridable by QDRANT_URL.

If none of the default files (sai-cfg.yml, sai-cfg.yaml, legacy indexer.yaml) exists, SAI prints note: no config at sai-cfg.yml — using built-in defaults (only hard dirs pruned) and continues with the defaults below. If you pass an explicit --config to a missing file, it is a hard error instead.

For how honor_noindex_marker / honor_noduplicate_marker interact with in-source sai-noindexing / sai-noduplicate comments, see opt-out markers. For the CLI flags that override these keys, see the CLI reference.

Key reference

Type, default, and resolution for every recognized key. “Has CLI flag” means a --flag can override it; “config-only” means the YAML key is the only way to set it (the value otherwise comes from a default or auto-detection).

Backend and embedder

backend selects the vector store: qdrant (default) or duckdb. embedder is the single knob for where and how embeddings are produced. Its default is backend-aware: the qdrant backend defaults to embedder: qdrant (server-side inference); the duckdb backend defaults to embedder: ort (local ONNX).

embedder accepts three values:

• ort — local ONNX Runtime; downloads the model from duckdb.model_repo. The default on the duckdb backend.
• ollama — remote Ollama HTTP server (see ollama).
• qdrant — Qdrant Cloud server-side inference (the Document API). Only valid with backend: qdrant (a duckdb backend with embedder: qdrant is a config error); it is that backend’s default.

How the two combine:

• backend: qdrant, embedder: qdrant (default) — code/queries are sent as Documents and the cluster embeds them. Requires Cloud (or an inference-enabled deployment) with model enabled in the Inference tab. This is the unchanged, pre-existing behavior.
• backend: qdrant, embedder: ort/ollama — embed on-device and upsert raw vectors. Works against self-hosted / OSS Qdrant (no Inference engine, no API key needed) and any local model (e.g. jinaai/jina-embeddings-v2-base-code, 768-d). Requires a binary built with --features qdrant,ort (or qdrant,ollama); selecting a local embedder without that feature fails with a clear rebuild hint. vector_dim must match the chosen model exactly (validated at runtime).
• backend: duckdb, embedder: ort/ollama — embed locally and store vectors in a DuckDB file with a VSS/HNSW index.

DuckDB and the local embedders are feature-gated — the binary must be built with --features ort, --features ollama, or --features all.

See Qdrant Cloud → Local-embed mode.

Chunker

chunker is lines or ast. The default is smart: when no chunker is set on the CLI or in config, SAI selects ast if the binary was built with --features ast and any requested extension is in the AST-preferred set (ts, tsx, rs, go); otherwise it falls back to lines. The chunker still dispatches per file, so a mixed walk AST-parses files with a grammar and line-chunks the rest.

chunker: ast   # or: lines — CLI --chunker always wins

Model, vector dimension, and chunk size

model is the embedding model label (and, for Qdrant, must match the cluster’s Inference tab). Its default depends on the resolved embedder:

vector_dim must equal the embedder model’s output dimensionality. It is config-only and runtime-validated — a mismatch is a clear error. If you change model to one with a different dimensionality, set vector_dim to match (e.g. mxbai-embed-large = 1024, nomic-embed-text = 768).

max_chunk_chars is the character cap both chunkers honor (a ~4-chars/token approximation of the model’s window). When unset, the default is model-aware (default_cap in src/config.rs):

prefix_style (config-only)

prefix_style controls the embedding prefix policy and has no CLI flag. Accepted values are e5, qwen, and none. When omitted, it is auto-detected from the model name: a name containing e5 → e5, containing qwen → qwen, otherwise → none. It is applied by both local embedders and the Qdrant document path.

# prefix_style: e5   # e5 | qwen | none — omit to auto-detect from model

duckdb

Used only when backend: duckdb; ignored by Qdrant.

duckdb:
  path: .index/code.duckdb        # DuckDB file; created on first index
  # model_repo: jinaai/jina-embeddings-v2-base-code   # ort: HF repo for model.onnx + tokenizer.json
  # model_cache: .model_cache      # ort: offline ONNX/HF cache dir (unset by default)

duckdb.model_repo defaults to jinaai/jina-embeddings-v2-base-code for the ort embedder, and to Xenova/multilingual-e5-small otherwise. duckdb.model_cache is unset by default.

ollama

Used only when embedder: ollama.

ollama:
  # url: http://localhost:11434    # default
  # model: mxbai-embed-large       # default (1024-d → set vector_dim: 1024)

ollama.model defaults to mxbai-embed-large (1024-d). Set vector_dim to match the model you pull.

File selection: exclude_dirs, include, exclude

exclude_dirs:        # directory NAMES pruned during the walk (never descended into)
  - __tests__

include: []          # allow-list globs; empty = consider everything

exclude:             # glob patterns matched per file path; '**' spans directories
  - "**/*.test.ts"
  - "**/*.d.ts"

• exclude_dirs is additive — its entries are merged on top of the always-pruned dirs (see below). Listing a hard-pruned dir here is harmless.
• include is an allow-list of globs. When non-empty it becomes active: a file must match at least one include glob to be considered. When empty/omitted, everything is considered.
• exclude globs are matched against each file path. Exclude always wins over include — the gate is (!include_active || include matches) && !exclude matches.

Selection order (per file)

For each file under the root, after directory pruning and the extension filter:

1. include — if active, the file must match an include glob, else it is skipped.
2. exclude — if any exclude glob matches, the file is skipped (exclude wins over include).
3. skip_generated_marker, then strip_comments are applied to the surviving content.

skip_generated_marker (defaults to false)

skip_generated_marker is a plain bool — unlike most toggles it has no “absent = true” fallback. When the key is omitted, it defaults to false (generated-marker scanning is off). The shipped sai-cfg.yml sets it to true explicitly:

skip_generated_marker: true   # scan file head for @generated / "DO NOT EDIT." markers

When enabled, it skips files whose head carries an autogenerated marker (e.g. @generated, Go’s // Code generated ... DO NOT EDIT.) regardless of filename.

strip_comments

strip_comments defaults to true when omitted. It removes // and /* */ comments from C-family source before embedding so only code reaches the backend; string/template literals are preserved and line numbers stay accurate.

strip_comments: true

honor_noindex_marker / honor_noduplicate_marker

Both default to true when omitted. They are not present in the shipped sai-cfg.yml — they take effect via the defaults. See opt-out markers for the in-source sai-noindexing / sai-noduplicate behavior.

honor_noindex_marker: true        # respect sai-noindexing comments (skip chunk entirely)
honor_noduplicate_marker: true    # respect sai-noduplicate comments (index, but no clustering)

similarity

Thresholds for the sai_find_similar and sai_find_duplicates MCP tools and the similar / duplicates CLI subcommands. Per-knob resolution is CLI flag / MCP tool arg > config value > built-in default. All fields are optional. These cutoffs are model-specific — tune them per embedder.

similarity:
  find_similar_min_score: 0.85       # default 0.85 — drop neighbors below this cosine
  duplicate_min_score: 0.93          # default 0.93 — edge cutoff between two chunks
  duplicate_min_cluster_size: 2      # default 2    — smallest cluster to report
  top_k: 10                          # default 10   — nearest-neighbor fan-out per chunk

logging

Diagnostic-logging defaults. All diagnostics go to stderr (stdout stays JSON-RPC / data only — see the CLI logging reference). This block is the lowest-precedence tier: the RUST_LOG env var and the CLI flags (-v/--silent/--log-format/--timing) always override it. All fields are optional.

logging:
  level: info       # error | warn | info | debug | trace. Unknown → info. -v/-vv/--silent and RUST_LOG override.
  format: pretty    # pretty (human) | json (one object per line). --log-format overrides.
  timing: false     # per-operation timing spans (index/embed/query/sync durations). --timing on; --silent off.

Its main audience is the MCP server: the client launches the binary with a fixed command and environment, so setting RUST_LOG per client is awkward — a logging: block in the project’s sai-cfg.yml (which the server already reads) raises verbosity project-wide with no launch-config edits. For ordinary CLI runs, -v/RUST_LOG are usually more convenient and this block can be omitted entirely.

timing toggles only the per-operation span-close durations. The single end-of-command summary line (done … in Ns) is a plain info event and is unaffected — it prints whenever the level is info or lower, regardless of timing. For the precedence details and examples, see the CLI logging reference.

The config is read before the logging subscriber is installed, so a missing or malformed file at that point falls back to these defaults silently (the usual “no config” warning and hard errors on a bad explicit --config path still surface later during the normal config load).

Always-pruned directories

Independent of config, these directories are always pruned during the walk (the HARD_PRUNE_DIRS set in src/config.rs):

node_modules   .git   dist   build   target   .next   coverage   .turbo

exclude_dirs entries are added on top of this set; you cannot un-prune a hard-pruned dir via config.

AST-preferred extensions

The smart chunker default selects ast (when the ast feature is compiled in and no chunker was set explicitly) for these extensions (AST_PREFERRED_EXTS):

ts   tsx   rs   go

Any other extension falls back to the lines chunker even when ast support is present.

Environment variables

The Qdrant API key is a secret and is read only from the environment — it never belongs in YAML. The cluster URL is not secret: set it in YAML as qdrant.url, or via the QDRANT_URL environment variable (which takes precedence over the YAML value).

If your sai-cfg.yml contains qdrant.url it is safe to commit (the URL is not a secret); the key never lives there.

Footgun: unknown keys are silently ignored

The Config struct is deserialized with all fields optional and #[serde(default)], so a partial file still parses — and unrecognized or misspelled YAML keys are silently ignored. A typo like skip_generated_marekr: true or exclude_dir: will not raise an error; the intended setting never takes effect and the default is used instead. Double-check key spelling and nesting (e.g. duckdb.path, similarity.top_k) against the Key reference table above.

See also

• CLI reference — flags that override these keys
• Opt-out markers — sai-noindexing / sai-noduplicate detail

MCP server and tools

semanticastindexer mcp runs an MCP server over stdio, exposing SAI’s semantic code search to agentic coding tools (Claude Code, Cursor, Windsurf, Codex, and others). It is built on the official Rust MCP SDK (rmcp, Cargo feature mcp).

This page is the authoritative reference for the server and every tool it exposes. The .agents/skills/sai/SKILL.md agent skill defers to this page for the canonical tool contracts.

For per-client wiring (Claude Code, Cursor, Windsurf, Codex, …) see MCP clients. For the exact JSON shapes each tool returns, see Output schemas.

Server behavior

• Read-only by default. Search exposes no index/upsert/flush tools. Only sai_refresh and sai_sync write, and only when explicitly enabled (see below).
• Backend, embedder, and collection come from sai-cfg.yml. See Backends and embedders.
• Backend built once at startup. The backend and embedder are constructed a single time and reused across every tool call.
• Tools are sai_-prefixed so they stand apart from other MCP servers’ tools in the agent’s tool list.

Worker-thread model

The DuckDB backend is !Send/!Sync, but rmcp’s tool-handler futures must be Send. So the backend lives on a dedicated worker thread, and the server holds only a Send + Sync channel handle (BackendHandle). Each tool handler builds a request, sends it to the worker, and awaits a oneshot reply — keeping handler futures Send while serializing all backend access through the single-threaded connection. For the full rationale see how it works.

Concurrency and the index lock

The server opens the DuckDB index once and holds it for the life of the connection — read-only by default, or read-write under --allow-write. DuckDB enforces single-writer / multi-reader access at the file level across processes, so while an MCP server is connected:

• A separate CLI sync, full re-index, or flush (all open the database read-write) against the same .index will fail or block on the lock — even a read-only MCP connection blocks a CLI writer.
• Two read-only consumers (e.g. an MCP server plus a CLI duplicates/similar) coexist fine.

So the in-process write tools are the preferred way to keep the index fresh without shelling out: call sai_sync (reconcile the git-changed set) or sai_refresh (specific paths) on the running --allow-write server — they mutate through the same connection, so there is no second handle and no lock contention. If you do need heavy CLI maintenance, disable/disconnect the MCP server first (via your client’s MCP toggle), run the CLI command, then reconnect.

Threshold and limit resolution

Two resolution rules apply throughout the tools:

• Threshold resolution (per knob): MCP tool arg > config value > built-in default. The config values come from the similarity: block of sai-cfg.yml; the built-in defaults are find_similar_min_score = 0.85, duplicate_min_score = 0.93, duplicate_min_cluster_size = 2, and top_k = 10. These cutoffs are model-specific — tune them per embedding model. See Tuning similarity.
• Limit clamping: any caller-supplied limit / top_k is clamped to [1, 50] so a single call can’t request the world.

Tools

The server exposes seven tools. Four are read-only; sai_refresh and sai_sync write and sai_prepare_mcp_setup can execute a setup script — all gated behind explicit flags.

Skills & subagents that drive these tools

The MCP server gives an agent the capability; two companion artifacts (both installed by mcp-setup/setup.sh) give it the judgment to use the tools well:

• sai-deslop (.agents/skills/sai-deslop/SKILL.md) — a portable Agent Skill (Claude Code, Cursor, Windsurf, …) covering when to call each tool while coding and a triage protocol that forces the agent to read each finding’s real source and classify it (real / boilerplate / intentional / fragment) before proposing a verified consolidation. It treats a cluster as a hypothesis, not a conclusion.
• dedup-auditor (.claude/agents/dedup-auditor.md) — a Claude Code subagent that runs the heavyweight repo-wide sai_find_duplicates sweep and per-member reads in an isolated context, applies the same triage protocol, and returns a classified digest instead of a raw cluster dump.

Skills are the portable, behavior-shaping layer; the subagent is the context-isolating worker for repo-wide audits. Both ultimately call the sai_ tools below.

sai_search_code

General semantic search over the indexed code. The query is embedded as a query and the nearest indexed chunks are returned. The server over-fetches (about limit × 4, still clamped) so the language / path_glob post-filters can still return up to limit rows. When include_text is false, each snippet is capped to the first ~8 lines and ~800 chars.

sai_find_similar

Find code similar to either an inline code snippet or an existing indexed chunk addressed by path + line. Provide either code OR both path and line — not a mix:

• code is embedded as a passage (code-vs-code space) and requires a local embedder (the duckdb backend); calling it against a non-local-embedding backend returns an invalid_params error.
• path + line looks up the exact stored vector for that chunk (no re-embed) and excludes the chunk itself from its own results. If no indexed chunk exists at that location the call returns no indexed chunk at <path>:<line>.

min_score resolves as arg > config > built-in default 0.85. Omitting it still applies the configured (model-tuned) cut; pass an explicit 0.0 to see the raw score distribution.

sai_find_duplicates

Find near-duplicate clusters across the index. For each chunk it takes the chunk’s top_k nearest neighbours, keeps the edges whose similarity is >= min_score, and unions them into clusters via union-find. Clusters with size >= min_cluster_size are returned, largest first.

Note: min_score, min_cluster_size, and top_k follow arg > config > built-in default, while max_clusters is a fixed local default of 50 and has no config knob.

sai_index_status

Report index metadata for freshness and sanity checks. Takes no arguments. Returns the backend, collection, embedding model, vector dimension, total chunk count, and chunker.

sai_prepare_mcp_setup

Help an agent set up SAI as an MCP server for a project. By default it only returns the exact commands and an MCP config snippet to run; it executes the setup script only when execute: true and the server was started with --allow-setup. Without --allow-setup, an execute: true call is reported as blocked (Server not started with --allow-setup) and no script runs.

The recommended_command includes --target-dir and an explicitly-derived --features list. For a prebuilt/release binary — where mcp-setup/setup.sh is not on disk beside it — the command falls back to the install.sh one-liner instead, and an execute: true call is reported as blocked (building from source needs a source checkout).

The response includes the recommended setup command, an mcp_server_config_example, and a numbered next_steps list. The first build can take several minutes (much longer for ort).

sai_refresh

Write tool. Re-index specific files in place: for each path it deletes that path’s existing points, then re-chunks, re-embeds, and re-upserts the files that still exist and pass the index filters (extension, globs, not generated). Paths that are gone or excluded are removed. The whole batch runs in one bulk window (HNSW drop → per-path delete + re-embed + upsert → rebuild), reusing the same per-file logic as the sync command.

Gating: the server must be started with --allow-write. Without it the backend is opened read-only and any call returns:

server is read-only; restart with --allow-write to enable refresh

An empty paths array returns refresh requires at least one path; more than 200 paths returns too many paths (max 200). On success the tool returns the refreshed paths (with chunk counts) and the removed paths.

sai_sync

Write tool. Reconcile the index with the working tree, the MCP analog of the CLI sync command: it resolves the git-changed file set and runs them through the same per-file reconcile as sai_refresh (survivors re-chunked/re-embedded, deleted or now-excluded paths removed).

Gating: like sai_refresh, requires --allow-write (else server is read-only; restart with --allow-write to enable sync). When git finds nothing the tool returns an empty result with a note; more than 200 changed files returns an error suggesting a narrower since or explicit paths. On success it returns the same {refreshed, removed} shape as sai_refresh.

Wiring (.mcp.json)

Point command at the built binary (an absolute path is safest) and set cwd to the indexed project root so the server finds that project’s index and sai-cfg.yml:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"],
      "cwd": "/path/to/your/project"
    }
  }
}

Build with the needed features and index the project once before starting the server. To enable the write tool, add --allow-write to args; to allow sai_prepare_mcp_setup to execute its script, add --allow-setup:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml", "--allow-write"],
      "cwd": "/path/to/your/project"
    }
  }
}

See MCP clients for per-client config locations and the CLI reference for the full mcp flag list. For the response shape of each tool, see Output schemas.

Output schemas

This page is the precise, parseable contract for everything SAI returns: the structured JSON emitted by each MCP tool, and the plain-text lines printed by the CLI subcommands. Field names, nesting, and defaults are taken directly from the implementation (src/mcp.rs, src/search.rs, src/main.rs). For the tools and flags that produce these results, see the MCP server and tools reference and the CLI reference; for the meaning of terms like chunk, symbol, and cosine similarity, see the glossary.

All MCP tools return their payload as a structured result (the object shown in each section below is the structured content). Every numeric score / sim / min_sim / max_sim is a cosine similarity in roughly [-1, 1], with 1.0 being identical direction.

MCP tool: sai_search_code

Semantic search. The query is embedded and matched against indexed chunks; optional language / path_glob post-filters are applied, then results are truncated to the (clamped) limit.

{
  "hits": [
    {
      "path": "src/auth/session.ts",
      "start_line": 42,
      "end_line": 88,
      "symbol": "createSession",
      "score": 0.8123,
      "snippet": "export function createSession(user: User): Session {\n  // ...\n}"
    }
  ]
}

MCP tool: sai_find_similar

Neighbours of a code snippet (embedded as a passage) or of an existing indexed chunk located by path + line (its stored vector is reused and the chunk excludes itself). Returns the same hit shape as sai_search_code.

{
  "hits": [
    {
      "path": "src/auth/legacy_session.ts",
      "start_line": 10,
      "end_line": 55,
      "symbol": "makeSession",
      "score": 0.9412,
      "snippet": "function makeSession(u) {\n  // ...\n}"
    }
  ]
}

Differences from sai_search_code:

• snippet is always capped (~8 lines / ~800 chars); sai_find_similar has no include_text option.
• Results are filtered by min_score before being returned. When min_score is omitted, the configured find_similar_min_score default applies; pass an explicit min_score: 0.0 to see the raw, unfiltered distribution.
• symbol is null for line-chunked indexes, same as above.

MCP tool: sai_find_duplicates

Codebase-wide near-duplicate clusters. For each chunk, its top-k neighbours are taken; edges with similarity >= min_score union chunks into clusters; clusters with >= min_cluster_size members are returned largest-first (tie-break: higher max_sim), truncated to max_clusters.

{
  "clusters": [
    {
      "size": 3,
      "members": [
        { "path": "src/a.ts", "start_line": 1,  "end_line": 40, "symbol": "parse" },
        { "path": "src/b.ts", "start_line": 12, "end_line": 51, "symbol": "parseInput" },
        { "path": "src/c.ts", "start_line": 5,  "end_line": 44, "symbol": null }
      ],
      "min_sim": 0.9512,
      "max_sim": 0.9987
    }
  ]
}

MCP tool: sai_index_status

Index metadata for freshness / sanity checks. Takes no arguments.

{
  "backend": "duckdb",
  "collection": "source_code",
  "model": "nomic-embed-text",
  "vector_dim": 768,
  "chunk_count": 1842,
  "chunker": "lines"
}

MCP tool: sai_refresh

Re-index specific files in place (write tool; requires the server to be started with --allow-write). Each path’s existing points are deleted first; paths that still exist and pass the index filters are re-chunked, re-embedded, and upserted. Gone or excluded paths are reported under removed.

{
  "refreshed": [
    { "path": "src/a.ts", "chunks": 7 },
    { "path": "src/b.ts", "chunks": 3 }
  ],
  "removed": [
    "src/deleted.ts"
  ]
}

MCP tool: sai_prepare_mcp_setup

Returns ready-to-run commands and an MCP server config block. With execute: true and the server started with --allow-setup, it additionally attempts to run the setup script and adds execution fields.

{
  "target_directory": "/path/to/project",
  "recommended_command": "/path/to/mcp-setup/setup.sh --non-interactive --backend duckdb --embedder ollama --target-dir \"/path/to/project\" --features \"mcp,duckdb,ollama\"",
  "mcp_server_config_example": {
    "mcpServers": {
      "sai": {
        "command": "<path-to-semanticastindexer>",
        "args": ["mcp", "--config", "sai-cfg.yml"],
        "cwd": "/path/to/project"
      }
    }
  },
  "next_steps": [
    "1. Run the recommended_command in a terminal (it can take 5-20 minutes the first time).",
    "2. After it finishes, index your project: cd <your-project> && <binary> --dry-run",
    "3. Then run without --dry-run to actually build the index.",
    "4. Add the mcp_server_config_example to your agent's MCP settings.",
    "5. Restart your agentic tool."
  ],
  "notes": "For fully offline use, prefer embedder=ort (much longer first build). The setup script was found next to this binary and the recommended_command is ready to run."
}

Conditional execution fields (added only when execute: true was requested):

CLI text output

The CLI prints human-readable lines to stdout (informational/progress messages go to stderr). Numeric similarity values are formatted to 4 decimal places. Indented result lines are prefixed with two spaces.

--query (semantic search)

A header line followed by one score path:start-end line per hit:

top 8 for: hash a password

  0.8421  src/auth/hash.ts:12-37
  0.7993  src/auth/verify.ts:5-29

• Header: top <N> for: <query> (N = number of hits).
• Each hit row: two leading spaces, then <score> <path>:<start_line>-<end_line>.
• The query path prints no symbol column.

duplicates

A summary line, then one block per cluster:

2 near-duplicate cluster(s) (min_score 0.93, min_cluster_size 2, top_k 8):
cluster (size 3, sim 0.9512..0.9987):
  src/a.ts:1-40  parse
  src/b.ts:12-51  parseInput
  src/c.ts:5-44  
cluster (size 2, sim 0.9301..0.9442):
  src/x.ts:3-20  
  src/y.ts:7-24  

• Summary: <N> near-duplicate cluster(s) (min_score <m>, min_cluster_size <c>, top_k <k>):.
• Cluster header: cluster (size <S>, sim <min_sim>..<max_sim>):.
• Member rows: two leading spaces, then <path>:<start_line>-<end_line> <symbol>.
• When nothing qualifies, a single line is printed instead: no near-duplicate clusters (min_score <m>, min_cluster_size <c>, top_k <k>).

similar

A summary line, then one row per neighbour:

3 similar (min_score 0.85):
  0.9412  src/auth/legacy_session.ts:10-55  makeSession
  0.8810  src/auth/session.ts:42-88  createSession
  0.8602  src/auth/token.ts:1-30  

• Summary: <N> similar (min_score <m>):.
• Each row: two leading spaces, then <score> <path>:<start_line>-<end_line> <symbol>.

Note on the symbol column

In the duplicates and similar text output, symbol is the last field on each member/hit row. For line-chunked indexes (the lines chunker) there is no symbol, so that field renders as an empty string — the row ends right after the line range plus the two trailing spaces (shown above for src/c.ts, src/x.ts, src/y.ts, and src/auth/token.ts). This is the text-output counterpart of symbol: null in the JSON schemas.

Backends and embedders

SAI (semanticastindexer) has a pluggable vector backend, and — for the DuckDB backend — a pluggable embedder. The backend decides where vectors are stored and how nearest-neighbour search runs; the embedder decides how text is turned into vectors.

The same five MCP tools and the same CLI subcommands work over either backend: sai_search_code, sai_find_similar, sai_find_duplicates, sai_index_status, sai_refresh (plus sai_prepare_mcp_setup). See the MCP server and tools page for the tool surface and the CLI reference for the equivalent subcommands.

Backends

Select the backend in sai-cfg.yml (backend: qdrant | duckdb) or override per run with --backend <name>. The backend factory is feature-gated: selecting a backend whose Cargo feature was not compiled in fails with a clear, actionable error, e.g.:

backend 'qdrant' selected but this binary was built without the 'qdrant' feature (rebuild with --features qdrant)

An unknown backend name fails with unknown backend '<name>' (expected 'qdrant' or 'duckdb').

qdrant — server-side or local embedding

The embedder field (default qdrant for this backend) decides where embeddings are produced; the storage and search paths are identical either way (a plain dense cosine collection).

embedder: qdrant (default). The backend never loads a model locally. Stored code and queries are sent as Document::new(text, model) and the cluster produces the embedding. The backend:

• Creates the collection (when missing) with VectorParams(vector_dim, Distance::Cosine) and a keyword payload index on path (so delete-by-path during sync is fast).
• Upserts chunks as passage:-prefixed Documents in batches of 32 (server-side inference runs per request).
• Queries with a query:-prefixed Document; query_by_vector over-fetches 8x and dedups by point id before truncating.
• begin_bulk / end_bulk are no-ops (there is no local index to drop and rebuild).
• Validates an existing collection’s vector dimension on open; a mismatch errors and tells you to re-run with --recreate (or delete the collection in the Qdrant Cloud UI).

embedder: ort/ollama. The backend embeds on-device with the configured ort/ollama embedder (exactly like the DuckDB path) and upserts raw Vec<f32> points — no Document, no server-side inference. The payload is byte-identical to the server path; only the vector source differs. The query side embeds locally too and reuses the same raw-vector NN path (query_by_vector). This makes the qdrant backend work against self-hosted / OSS Qdrant and lets you use any local model (e.g. the code-trained jinaai/jina-embeddings-v2-base-code, 768-d) without Cloud billing. Requires a binary built with --features qdrant,ort (or qdrant,ollama); selecting a local embedder without that feature fails with a clear rebuild hint. Walkthrough: Qdrant Cloud → Local-embed mode.

ℹ️ Plain OSS/local Qdrant has no inference engine — the Document API (embedder: qdrant) only works against Qdrant Cloud (or an inference-enabled deployment). To run against self-hosted/OSS Qdrant, use embedder: ort/ollama (above), which embeds on-device and never calls the Document API. See Qdrant Cloud for setup.

duckdb — local VSS/HNSW

The DuckDB backend persists everything to a single file (duckdb.path, e.g. .index/code.duckdb). On open it loads the VSS extension and enables HNSW persistence:

• The collection table stores embedding FLOAT[vector_dim] plus the chunk metadata (id, path, language, start_line, end_line, text, symbol, commit_sha, dirty, no_duplicate).
• The HNSW index is created USING HNSW(embedding) WITH (metric='cosine'). Search uses array_cosine_distance and returns score = 1 - distance (higher is better, matching Qdrant’s cosine score).
• HNSW can return the same id more than once, so query_by_vector over-fetches 8x, dedups by id, then truncates to the limit.
• A writable open sets SET hnsw_enable_experimental_persistence = true so the index survives across close/reopen. The MCP server opens the file read-only and does not enable persistence writes (a read-only handle must not mutate the DB).
• The VSS extension is loaded with a pure LOAD vss; first (works on read-only handles if VSS was already installed by any process), then INSTALL vss; LOAD vss;, then the community repo. If none succeed you get an actionable error; pre-install once with a writable run or duckdb -c "INSTALL vss;". See Troubleshooting and FAQ.

ℹ️ DuckDB sync recall note. The DuckDB VSS HNSW index loses recall after in-place deletes, so sync drops and recreates the index around its changed-file loop (begin_bulk drops the index, end_bulk recreates it) — effectively a full rebuild of the HNSW graph. This is correct but means a DuckDB sync is not as cheap as Qdrant’s, where begin_bulk/end_bulk are no-ops. (DELETE alone does not trigger an HNSW rebuild, so delete_by_path needs no index teardown.)

DuckDB embedders

The DuckDB backend produces vectors locally via a pluggable embedder (embedder: ort | ollama in sai-cfg.yml, or --embedder <name>). ort is the default.

Like backends, the embedder is feature-gated: selecting an embedder whose Cargo feature was not compiled in fails with embedder '<name>' selected but this binary was built without the '<name>' feature (rebuild with --features <name>). An unknown embedder name fails with unknown embedder '<name>' (expected 'ort' or 'ollama').

The ort pipeline

For each batch the on-device ONNX embedder runs this exact sequence:

1. Prefix — apply the resolved prefix policy (format_passage / format_query); see Embedding prefixes below.
2. Tokenize — pad/truncate to 512 tokens (MAX_TOKENS), padding BatchLongest so every row in a batch is the same length (ONNX needs rectangular tensors).
3. Run ONNX — feed input_ids + attention_mask (and a zeroed token_type_ids iff the loaded model declares that input) and read last_hidden_state [batch, seq, hidden]. If the export names the first output differently, the first output by index is used as a fallback.
4. Mean-pool over the attention mask — sum hidden states weighted by the mask, divide by the mask sum (so padding tokens contribute nothing).
5. L2-normalize — divide each pooled vector by its L2 norm, so cosine similarity is a plain dot product.

Batches are sized at 32 passages per forward pass and length-sorted before batching (then scattered back to caller order) so one long passage does not inflate a whole batch with padding. Inference is synchronous CPU work sized to available_parallelism() intra-op threads — acceptable for a one-shot CLI batch job.

The ollama embedder

The Ollama embedder POSTs prefixed inputs to {ollama.url}/api/embed and reads { "embeddings": [[...], ...] }. Requirements:

• ollama.model is required — there is no default (Ollama embed models vary). Set it to an embed-capable model. Construction fails clearly if it is unset: embedder 'ollama' selected but ollama.model is unset — set ollama.model … (e.g. nomic-embed-text).
• ollama.url defaults to http://localhost:11434 upstream; a trailing / is trimmed.
• Start the server (ollama serve) and pull the model (ollama pull nomic-embed-text). A non-success HTTP status produces an error that suggests ollama pull <model>; a connection failure asks whether ollama serve is running.

See Ollama for end-to-end setup.

vector_dim must match the model

vector_dim is validated at runtime and must equal the embedder’s output dimension — the DuckDB column is literally FLOAT[vector_dim]. A produced-vs-configured mismatch is a clear error:

embedder produced 768-d vectors but vector_dim=384 — set vector_dim to match the model
(e5-small=384, nomic-embed-text=768, mxbai-embed-large=1024)

There are two layers of this check:

• On open (DuckDB), if the table already exists, the embedding column type is compared against FLOAT[vector_dim]. A mismatch is a typed DimMismatch error carrying the DuckDB file path, so the CLI can offer an interactive delete-and-rebuild instead of string-matching. The message names the actual vs expected type and tells you to delete the file or re-index with --recreate. Qdrant performs the equivalent check against the collection’s configured vector size on open.
• Per embedding (check_dim), every produced query/passage vector is checked before it hits the index.

Reference dimensions: e5-small = 384, nomic-embed-text = 768, mxbai-embed-large = 1024, jina-embeddings-v2-base-code = 768.

⚠️ Changing vector_dim (or the model) requires a fresh index. Delete the DuckDB file (e.g. .index/code.duckdb) or re-index with --recreate.

Embedding prefixes

A model-aware prefix policy is resolved once when the plan is built (explicit prefix_style config wins; otherwise it is auto-detected from the model name) and applied by both embedders and the Qdrant Document path through one shared pair of helpers.

⚠️ E5 passage/query asymmetry caveat. The E5 family is trained with the asymmetric passage: / query: scheme — both embedders and Qdrant apply it when prefix_style: e5. A non-E5 model (e.g. many Ollama models, or a symmetric code model) may want different (or no) prefixes; relevance can suffer if it was not trained with this asymmetric scheme. Set prefix_style: none (or qwen) to match the model. See Chunking → embedding prefixes for the chunk-side picture.

Offline / cached ort

For air-gapped or repeatable runs, point duckdb.model_cache at a pre-populated HuggingFace cache directory. The ort embedder passes it through as the hf-hub cache dir, so it reuses onnx/model.onnx and tokenizer.json from disk instead of fetching them. If the download fails, the error suggests exactly this: … (check network or set duckdb.model_cache to a pre-populated dir).

🔒 Qdrant creds stay in the environment. QDRANT_URL / QDRANT_API_KEY are read from the environment, never from YAML. See Environment variables and Security and privacy.

Recommended model for code de-duplication

e5-small is a multilingual text model: distinct functions in the same language all embed ~0.91 cosine-similar, so sai_find_duplicates collapses into one giant cluster at any threshold. A code-trained embedder spreads functions far apart and surfaces real near-duplicates (even across different names). Recommended drop-in (stays on the offline ort path):

model: jinaai/jina-embeddings-v2-base-code   # 161M, code-trained (CodeSearchNet)
vector_dim: 768                              # MUST match the model
prefix_style: none                           # symmetric model — no passage:/query: prefix
duckdb:
  model_repo: jinaai/jina-embeddings-v2-base-code   # ort downloads onnx/model.onnx + tokenizer.json
similarity:
  duplicate_min_score: 0.88                  # code models run LOWER than e5 (no mega-cluster at any threshold)

See Choosing a model for the full comparison and Tuning similarity for thresholds.

⚠️ First-run download caveat (hf-hub 0.3 + Xet). The pinned hf-hub (0.3) fails to fetch tokenizer.json from this repo because it is on HuggingFace Xet storage (relative URL without a base). Until hf-hub is upgraded, stage the tokenizer once into the HF cache:

SNAP=~/.cache/huggingface/hub/models--jinaai--jina-embeddings-v2-base-code/snapshots/*/
curl -sL https://huggingface.co/jinaai/jina-embeddings-v2-base-code/resolve/main/tokenizer.json -o $SNAP/tokenizer.json

The onnx/model.onnx download works; only tokenizer.json needs staging. After that, normal runs use the cache — no HF_HUB_OFFLINE needed. Changing vector_dim requires a fresh index: delete .index/code.duckdb (or run with --recreate). More fixes live in Troubleshooting and FAQ.

Qdrant requirements

These apply to embedder: qdrant (Qdrant Cloud server-side inference). With embedder: ort/ollama you need neither Cloud Inference nor an API key — any reachable Qdrant (including a local docker run qdrant/qdrant) works, since embedding happens on-device; just set QDRANT_URL to the gRPC endpoint (:6334).

• A Qdrant Cloud cluster with Inference enabled and the intfloat/multilingual-e5-small model available (Cluster → Inference tab). Vector size 384, context window 512 tokens.
• Credentials via the environment (never hard-coded):

export QDRANT_URL="https://<cluster-id>.<region>.aws.cloud.qdrant.io:6334"   # gRPC port :6334
export QDRANT_API_KEY="<key from the cluster's API Keys tab>"

If QDRANT_API_KEY is unset, the backend warns that Qdrant Cloud will reject the request. If a server-side upsert fails it asks whether Inference is enabled on the cluster. Full walkthrough: Qdrant Cloud.

How the backend is selected

The factory reads backend from the resolved plan and opens the DuckDB arm per an Access mode:

• ReadWrite — normal open with index maintenance, writes, and HNSW persistence (indexing, refresh, sync).
• ReadOnly — search-only path used by the MCP server and the CLI similar / duplicates subcommands. The DuckDB file must already exist (a missing index is an actionable error, since read-only search never indexes).

Qdrant is a remote, already-read-capable path, so both access modes behave identically there.

See also

• Choosing a model — model trade-offs and recommendations.
• Tuning similarity — thresholds and scoring.
• Chunking — how source is sliced into embeddable chunks.
• Configuration — every sai-cfg.yml key.
• MCP server and tools — sai_-prefixed tools over either backend.
• Qdrant Cloud and Ollama — backend/embedder setup.
• Troubleshooting and FAQ — VSS, downloads, dim mismatches.
• Glossary — terms used above.

Chunking

How SAI splits each source file into the embeddable units (“chunks”) that get vectorized, stored, and compared. SAI ships two chunkers — the line-window chunker (lines) and the symbol-aware tree-sitter chunker (ast) — and picks one for you unless you say otherwise.

For the terms used here (chunk, symbol, embedder), see the glossary. To choose the chunker on the command line or in sai-cfg.yml, see the CLI reference and Configuration.

Smart default (lines vs ast)

When you do not explicitly set --chunker (CLI) or chunker: (in sai-cfg.yml), SAI defaults as follows:

• Languages with good AST support — currently ts, tsx, rs, go, and py — and a binary built with --features ast default to the symbol-aware ast chunker.
• Everything else defaults to the reliable lines chunker.

You can force a specific chunker:

# Force the line-window chunker
sai index --chunker lines

# Force the AST chunker (requires a binary built with --features ast)
sai index --chunker ast

# sai-cfg.yml
chunker: ast   # or: lines

The AST chunker is function-only

The index exists to compare functions for near-duplicates, so the AST chunker (tree-sitter, for TypeScript/TSX + Rust + Go + Python) deliberately embeds functions and nothing else. Earlier versions emitted non-function nodes too, which flooded duplicate detection with tiny, near-identical vectors — making every run report the whole codebase as duplicated.

What gets chunked

One chunk per named function at any nesting depth. The function’s name is stored as the chunk’s symbol:

What is NOT chunked

The AST chunker drops everything that is not a function. A file made entirely of these produces zero chunks — nothing else is indexed:

• Classes, interfaces, and type aliases
• const / static / mod / struct / enum / trait items
• Imports and top-level statements
• Bare anonymous closures / func literals / Python lambdas (one-line lambdas are tiny and near-identical, so capturing them would collapse every duplicate run into one cluster)

Go’s only nested-function form is a func literal (a closure), and — like Rust closures — those are intentionally not captured.

Edge cases and fallbacks

• Oversized function — a function whose byte length exceeds max_chunk_chars is line-split over its own span via the shared line chunker, and every resulting window keeps the function’s symbol.
• No functions in the file — the file produces no chunks; nothing else is indexed.
• Parse-failure fallback — a file that fails to parse (root parse error), or any extension without a tree-sitter grammar (anything other than .ts / .tsx / .rs / .go / .py, e.g. Java), silently falls back to the line chunker.
• Comments are stripped before chunking — so the AST parses comment-stripped text. Stripping preserves the exact line count, so a chunk’s start_line/end_line still point at the real lines in the original file.
• Exact-span dedupe — two captures sharing an identical byte span are collapsed to one chunk, preferring the one that carries a symbol. Nested functions are emitted in their own right and remain part of their enclosing function’s chunk (no carve-out); the overlap is acceptable for near-duplicate detection.

Feature gating

The ast chunker is feature-gated. If chunker: ast is explicitly selected — or auto-selected for a TS/TSX/Rust/Go/Python file — on a binary built without --features ast, SAI fails fast at startup with a clear, actionable error:

chunker 'ast' selected but this binary was built without the 'ast' feature (rebuild with --features ast)

Rebuild with the feature to enable it:

cargo build --release --features ast
# or pull in everything:
cargo build --release --features all

An unknown chunker value (anything other than lines or ast) is likewise rejected at startup:

unknown chunker '<value>' (expected 'lines' or 'ast')

Chunk-size cap (max_chunk_chars)

max_chunk_chars is the character bound that both chunkers honor. It is a ~4-characters-per-token approximation of the embedding model’s token window, so no tokenizer is needed at index time. The line chunker uses it to bound each window; the AST chunker uses it to decide when a function is oversized and must be line-split.

When max_chunk_chars is left unset, SAI picks a model-aware default:

For picking a model and embedder, see Choosing a model and Backends and embedders.

Embedding prefixes (prefix_style: e5 | qwen | none)

Some embedding models expect their input to carry a task prefix. SAI resolves the prefix style once and applies it through one shared helper used by both the local embedders and the Qdrant Document path, so passages and queries are prefixed identically everywhere.

Resolution order: an explicit prefix_style in config wins. Otherwise SAI auto-detects from the model name: a name containing e5 → E5; a name containing qwen → Qwen; anything else → None. So the Qwen prefix applies when the model name contains qwen, or when you set prefix_style: qwen explicitly.

Related pages

• Indexing a project — running the indexer end to end.
• Opt-out markers — sai-noindexing / sai-noduplicate markers, which are scanned per chunk over the original (pre-strip) lines.
• Configuration — chunker, max_chunk_chars, prefix_style, and strip_comments keys.
• Backends and embedders — how chunks are embedded and stored.

MCP clients

semanticastindexer mcp runs a Model Context Protocol (MCP) server over stdio, exposing the sai_-prefixed tools (sai_search_code, sai_find_similar, sai_find_duplicates, sai_index_status, sai_refresh) to any agentic coding client. This page is a per-client integration guide: where each client’s config file lives, the exact server snippet, the install.sh --platform <id> command that wires it, and how to restart, verify, and test-drive the connection.

For the tool reference (arguments, thresholds, write tool) see ../reference/mcp-server.md. For installing the binary first, see ../installation.md. If a client doesn’t list the tools after a restart, jump to ../operations/troubleshooting.md.

Before you start

1. Install the binary (../installation.md). Note its absolute path — an absolute command path is the safest choice in every client config below. When built from source the binary lands at ./target/release/semanticastindexer.
2. Index the project once before starting the server.

The server is read-only by default. The backend, embedder, and collection are read from sai-cfg.yml — edit that file to switch embedders or collections rather than passing flags here. The server’s cwd must be the indexed project root so it finds that project’s DuckDB index and sai-cfg.yml.

Using the installer to wire a client

On macOS/Linux the install script can do the wiring for you. Add --platform <id>:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform cursor

Supported ids: claude-code, claude-desktop, cursor, windsurf, continue, codex, hermes, ollama, generic.

• By default the installer prints the config snippet and the exact target path so you can paste it yourself.
• Pass --write to merge the snippet into your client’s config file (best-effort, with a backup).
• Run the installer without --platform and it can prompt interactively to pick a client.
• On Windows: install the binary first, then paste the printed block into your client’s config by hand.

The server entry is named sai in the configs the installer manages.

Claude Code

Config: project .mcp.json (in the repository you want to search), plus the sai skill installed into ~/.claude/skills/sai/.

The --platform claude-code install gives you the full skill experience — it wires the project .mcp.json and installs the skill:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform claude-code --write

Server snippet for .mcp.json:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"]
    }
  }
}

Reload: Claude Code loads .mcp.json from the project root on startup — restart the session (or reopen the project) to pick up changes.

Verify: the sai_ tools appear in the tool list; the skill prompt also references sai_search_code, sai_find_similar, and sai_find_duplicates.

First test query: ask Claude to “search the codebase for where authentication tokens are validated” — it should call sai_search_code.

Claude Desktop

Config: claude_desktop_config.json, per OS:

Install command:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform claude-desktop --write

Server snippet:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"],
      "cwd": "/absolute/path/to/your/indexed/project"
    }
  }
}

Claude Desktop has no per-project working directory, so set cwd explicitly to the indexed project root.

Restart: fully quit and relaunch the Claude Desktop app.

Verify: open the tools / MCP servers indicator and confirm the server is connected and the sai_ tools are listed.

First test query: “find functions similar to this one” with a pasted snippet — it should call sai_find_similar.

Cursor

Config: ~/.cursor/mcp.json (global), or project .cursor/mcp.json.

Install command:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform cursor --write

Server snippet:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"],
      "cwd": "/absolute/path/to/your/indexed/project"
    }
  }
}

Reload: open Cursor Settings → MCP and toggle the server, or restart Cursor.

Verify: the MCP settings panel shows the server connected with its tools enumerated.

First test query: “look for near-duplicate code across the repo” — it should call sai_find_duplicates.

Windsurf / Cascade

Config: ~/.codeium/windsurf/mcp_config.json.

Install command:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform windsurf --write

Server snippet:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"],
      "cwd": "/absolute/path/to/your/indexed/project"
    }
  }
}

Reload: in Cascade, open the MCP / plugins panel and press refresh, or restart Windsurf.

Verify: the Cascade MCP panel lists the server and its sai_ tools.

First test query: “what does the indexing pipeline do?” — it should call sai_search_code.

Continue.dev

Config: ~/.continue/config.yaml — add an mcpServers block (YAML).

Install command:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform continue --write

Server snippet (YAML):

mcpServers:
  - name: sai
    command: /path/to/semanticastindexer/target/release/semanticastindexer
    args:
      - mcp
      - --config
      - sai-cfg.yml
    cwd: /absolute/path/to/your/indexed/project

Reload: reload the Continue extension (or restart the IDE) so it re-reads config.yaml.

Verify: the Continue assistant’s tool list includes the sai_ tools.

First test query: “find where config defaults are resolved” — it should call sai_search_code.

Codex CLI

Config: ~/.codex/config.toml, under an [mcp_servers.sai] table (TOML).

Install command:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform codex --write

Server snippet (TOML):

[mcp_servers.sai]
command = "/path/to/semanticastindexer/target/release/semanticastindexer"
args = ["mcp", "--config", "sai-cfg.yml"]
cwd = "/absolute/path/to/your/indexed/project"

Reload: start a new Codex CLI session so it re-reads config.toml.

Verify: Codex enumerates the MCP server’s tools at session start; the sai_ tools should be present.

First test query: “search for the embedding model loader” — it should call sai_search_code.

Generic / manual client

Any stdio MCP client works. Use --platform generic to print a portable .mcp.json block and its target path:

curl -fsSL https://maadgrom.github.io/semanticastindexer/install.sh | bash -s -- --platform generic

Then paste the printed block into your client’s MCP config. The shape is the same mcpServers object used above:

{
  "mcpServers": {
    "sai": {
      "command": "/path/to/semanticastindexer/target/release/semanticastindexer",
      "args": ["mcp", "--config", "sai-cfg.yml"],
      "cwd": "/absolute/path/to/your/indexed/project"
    }
  }
}

The rules are the same for any client:

• command points at the binary (absolute path is safest).
• args starts with mcp; backend, embedder, and collection are read from sai-cfg.yml.
• cwd is the indexed project root.

Restart the client, confirm the sai_ tools load, and run a sai_search_code query as a smoke test.

Enabling the write tool

All snippets above start the server read-only. sai_refresh (re-index specific files) is a write tool and is only usable when the server is started with --allow-write — without it, the index is opened read-only and sai_refresh returns server is read-only; restart with --allow-write. To enable it, add the flag to args:

"args": ["mcp", "--config", "sai-cfg.yml", "--allow-write"]

See ../reference/mcp-server.md for what sai_refresh does and the similarity thresholds that govern sai_find_similar / sai_find_duplicates.

Verifying and troubleshooting

If a client starts but the sai_ tools don’t appear, or queries error out:

• Confirm the command path is correct and the binary was built with --features all.
• Confirm cwd is the project root you actually indexed (so the DuckDB index and sai-cfg.yml are found).
• Make sure you indexed the project once before starting the server.
• Fully restart the client after editing its config.

See ../operations/troubleshooting.md for more.

Qdrant Cloud

The qdrant backend stores vectors in a Qdrant Cloud collection and produces embeddings using Qdrant Cloud’s server-side inference — no embedding model runs locally. This page covers how to set up a cluster, what semanticastindexer (SAI) expects from it, and how the collection is created and validated.

For the full backend matrix (qdrant vs. duckdb), see backends and embedders. For how the connection URL and API key are configured, see Configuration → Environment variables.

How server-side inference works

With the qdrant backend, SAI never loads a tokenizer or ONNX model. Instead it hands raw text to Qdrant Cloud via the Document API: each stored chunk is sent as a Document (a text string plus a model name), and the cluster computes the vector inside the cluster. The same applies to searches — a query is sent as a Document, embedded server-side, then used for nearest-neighbour lookup.

Stored chunks are embedded as passage: <code> and queries as query: <text>, matching E5’s asymmetric prefix scheme.

⚠️ Server-side inference requires Qdrant Cloud (or an inference-enabled deployment). The Document API used by the default embedder: qdrant path does not exist in plain OSS / self-hosted Qdrant. If you point the qdrant backend at a vanilla local Qdrant without switching to local-embed mode, upserts and queries will fail. To use the qdrant backend with OSS / self-hosted Qdrant, set embedder: ort (or ollama) — see Local-embed mode below. To avoid running Qdrant at all, use the duckdb backend instead — see backends and embedders.

Cluster requirements

Set up a Qdrant Cloud cluster with:

• Inference enabled on the cluster.
• The intfloat/multilingual-e5-small model available — enable it under the Cluster → Inference tab.
• Vector size 384.
• Context window 512 tokens.

These match SAI’s expectations for the Qdrant path; chunks are tokenized within the 512-token context window server-side.

Connection

The cluster URL can live in sai-cfg.yml (qdrant.url) or come from the QDRANT_URL environment variable, which overrides the YAML value. The API key is a secret and is read only from QDRANT_API_KEY in the environment — never put it in YAML.

In sai-cfg.yml:

backend: qdrant
qdrant:
  url: https://<cluster-id>.<region>.aws.cloud.qdrant.io:6334   # gRPC port :6334

The API key always comes from the environment (optionally the URL too):

export QDRANT_API_KEY="<key from the cluster's API Keys tab>"
# optional: supply or override the URL from the environment instead of YAML
export QDRANT_URL="https://<cluster-id>.<region>.aws.cloud.qdrant.io:6334"

Notes from the connection code:

• A URL is required: set qdrant.url or QDRANT_URL. If neither is set, SAI errors telling you to provide the cluster gRPC endpoint (https://<id>.<region>.aws.cloud.qdrant.io:6334).
• Use the gRPC port :6334, not the REST port.
• If QDRANT_API_KEY is unset (or empty), SAI prints a warning and proceeds — Qdrant Cloud will then reject the request, so set the key.

Select the backend in sai-cfg.yml (backend: qdrant) or override per run with --backend qdrant. See the configuration reference and the CLI reference.

How the collection is created

The first time SAI runs against a missing collection, it creates one configured for the Qdrant inference path:

• Vector params: size = vector_dim (384), distance = Cosine.
• A keyword payload index on path, which makes the delete-by-path filter used during sync fast.

On success you’ll see output like:

created collection '<name>' (384 dims, cosine, path index)

If the collection already exists and you are not recreating it, SAI prints using existing collection '<name>' and validates the stored dimension (see below) before reusing it.

Dimension validation on reuse

When reusing an existing collection, SAI reads the collection’s configured vector dimension and compares it to the vector_dim of the current run. If they differ, the run fails fast with a message like:

Qdrant collection '<name>' has vector dimension <N> but this run uses vector_dim=<M>.
This usually means the embedding model was changed without recreating the collection.
Re-run with --recreate (or manually delete the collection in the Qdrant Cloud UI).

This catches the common mistake of pointing the indexer at an old collection after changing the embedding model / vector_dim. Without it, Qdrant would otherwise fail later with dimension-mismatch errors during upsert or query.

Changing models: one-time re-index

The collection’s vector size is fixed at creation. If you change the model (and thus vector_dim), you must recreate the collection — there is no in-place migration. Re-index once with --recreate:

# Drops the existing collection, recreates it with the new dims, and re-indexes.
sai index --backend qdrant --recreate

--recreate drops the existing collection (dropped existing collection '<name>') and creates a fresh one with the current vector params and the path payload index. Alternatively, delete the collection manually in the Qdrant Cloud UI and let the next run create it.

See the CLI reference for the index command and --recreate flag.

Local-embed mode (self-hosted / OSS Qdrant)

Setting embedder: ort (or ollama) switches the qdrant backend from Qdrant Cloud’s Document API to local embedding. (embedder: qdrant, the default for this backend, is the server-side path.) In local-embed mode SAI embeds code and queries locally (using the ort or ollama embedder) and upserts raw Vec<f32> vectors directly — no server-side inference engine is required. This unlocks self-hosted / OSS Qdrant instances, which have no inference engine, and lets code-trained models such as jinaai/jina-embeddings-v2-base-code run against a local cluster with no Cloud billing.

Configuration

backend: qdrant
embedder: ort            # ort or ollama — selects the local embedder (qdrant = server-side)
model: jinaai/jina-embeddings-v2-base-code
vector_dim: 768          # MUST match the model
prefix_style: none       # symmetric code model — no passage:/query: prefix
qdrant:
  url: http://localhost:6334   # gRPC port :6334

The embedder field is the single knob: qdrant (default) means Qdrant Cloud server-side inference; ort/ollama mean embed locally. There is no separate qdrant.inference knob.

Build requirement

Local-embed mode requires the ort or ollama Cargo feature:

cargo build --release --features qdrant,ort
# or, for Ollama:
cargo build --release --features qdrant,ollama

Selecting embedder: ort/ollama for the qdrant backend in a binary compiled without either feature fails with a clear error that tells you to rebuild with the appropriate feature flag.

OSS Qdrant notes

• Use the gRPC port :6334, not the REST port.
• No API key is needed for an unauthenticated OSS Qdrant instance — omit QDRANT_API_KEY.
• The duplicates sweep (sai_find_duplicates / duplicates CLI subcommand) was already inference-free and is unaffected by this setting; it always operates on raw stored vectors.
• To run OSS Qdrant locally: docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant.

Related pages

• Configuration → Environment variables — qdrant.url, QDRANT_URL, QDRANT_API_KEY.
• Backends and embedders — qdrant vs. duckdb.
• ../reference/configuration.md — backend, model, vector_dim.

Ollama

Ollama is a local HTTP server that runs embedding (and chat) models on your machine. SAI’s ollama embedder uses it as the embedding backend for the DuckDB vector store: instead of running an ONNX model in-process (the ort embedder), SAI POSTs your chunk text to a running Ollama server and stores the returned vectors in a local DuckDB file.

This is a good fit when an Ollama service is already running — for example a shared CI/CD box or a developer machine that hosts embedding models for several tools — so SAI doesn’t have to download or load a model itself.

The ollama embedder is only used by the duckdb backend. The qdrant backend does its embedding server-side (Qdrant Cloud inference) and ignores these settings. See Backends & embedders.

1. Run the Ollama server

ollama serve

By default the server listens on http://localhost:11434, which is also SAI’s default ollama.url.

2. Pull an embedding model

Pick an embedding-capable model and pull it. Two common choices:

# 1024-dimensional
ollama pull mxbai-embed-large

# 768-dimensional
ollama pull nomic-embed-text

Browse more options in Ollama’s embedding models.

3. Configure SAI

Select the duckdb backend with the ollama embedder and point it at your server. The critical part is matching vector_dim to the model’s output dimension — SAI validates this at runtime and fails with a clear error (e.g. embedder produced 768-d vectors but vector_dim=384 …) on a mismatch.

backend: duckdb
embedder: ollama
vector_dim: 1024            # MUST match the model (mxbai-embed-large = 1024)
ollama:
  url: http://localhost:11434   # default — omit if unchanged
  model: mxbai-embed-large

Using nomic-embed-text instead:

backend: duckdb
embedder: ollama
vector_dim: 768             # nomic-embed-text = 768
ollama:
  model: nomic-embed-text

Reference values:

Notes from the config defaults:

• ollama.url defaults to http://localhost:11434 — you only need to set it when Ollama runs elsewhere.
• ollama.model defaults to mxbai-embed-large (1024-d) when the ollama embedder is selected. If you set it to the empty string, SAI errors with a message telling you to set ollama.model.

You can also override the backend/embedder per run on the CLI instead of in YAML:

sai index . --backend duckdb --embedder ollama --vector-dim 1024

See Configuration and the CLI reference for the full set of keys and flags.

How it works

For each batch of chunks, SAI POSTs to the embed endpoint:

POST {ollama.url}/api/embed
{ "model": "<ollama.model>", "input": ["<text>", ...] }

and reads the embeddings back from the embeddings field of the JSON response. A trailing slash on ollama.url is trimmed, so both http://localhost:11434 and http://localhost:11434/ work. If the server is unreachable or the model isn’t pulled, SAI surfaces an actionable error (for example, suggesting ollama pull <model>).

The E5-prefix caveat

SAI was built around the E5 family of text embedders, which use asymmetric prefixes: indexed text gets a passage: prefix and search queries get a query: prefix. The ollama embedder applies the same prefix policy as the ort embedder — controlled by prefix_style — so by default it prepends passage:/query: to your inputs.

Most Ollama embedding models (mxbai-embed-large, nomic-embed-text, …) are not E5 models and were not trained with this asymmetric scheme, so the injected prefixes can hurt relevance. If your model isn’t an E5 model, set a symmetric (bare) prefix policy:

embedder: ollama
prefix_style: none          # don't prepend passage:/query:
ollama:
  model: nomic-embed-text
vector_dim: 768

prefix_style accepts e5, qwen, or none; when unset it is auto-detected from the model name. For the full explanation of when to keep or drop prefixes, see Choosing a model.

Good fit for CI/CD

Because the ollama embedder downloads nothing at index time — it just calls an HTTP endpoint — it pairs well with environments where an embedding service is already up. In CI/CD you can run ollama serve (with the model pre-pulled) and point every SAI job at it via ollama.url, keeping the indexing step fast and network-light. See CI/CD.

See also

• Backends & embedders — how ort vs ollama differ on the DuckDB backend.
• Configuration — the ollama: and similarity: keys.
• Choosing a model — picking a model and prefix policy.
• Ollama embedding models.

Troubleshooting and FAQ

This is the FAQ hub for semanticastindexer (SAI). Each entry pairs a symptom you’ll see (an error message, an empty result, a missing tool) with the cause and the fix. Most problems fall into one of three buckets: a first-run model download, a dimension/feature mismatch, or an MCP wiring issue.

For deeper background see Backends and embedders, Environment variables, and MCP clients.

Q: The first run hangs for a long time, or fails with no network. What is it doing?

On the local DuckDB path with the ort (ONNX Runtime) embedder, the very first run downloads the model and tokenizer from Hugging Face: onnx/model.onnx and tokenizer.json from the repo named by duckdb.model_repo. That download happens once; later runs reuse the cache.

• Where the cache lives: the standard Hugging Face Hub cache at ~/.cache/huggingface. Each repo lands under ~/.cache/huggingface/hub/models--<owner>--<name>/.
• Slow download: the ONNX model is the big file (hundreds of MB for code models). Progress is printed; let it finish — it is cached afterward.
• No / restricted network: if you already have a populated cache, point the embedder at it instead of the default location and run offline.

# sai-cfg.yml — reuse a pre-populated HF cache (no network on subsequent runs)
duckdb:
  model_cache: /path/to/huggingface/cache

duckdb.model_cache sets the Hugging Face cache directory the ort embedder reads from (it maps to the Hub client’s cache dir). To force fully offline behavior regardless of the cache location, export HF_HUB_OFFLINE:

export HF_HUB_OFFLINE=1

With HF_HUB_OFFLINE=1, the Hub client never reaches the network and uses only what’s already cached — so the model and tokenizer must already be present, or the run errors. If you pre-stage the cache once and then run normally, you do not need HF_HUB_OFFLINE at all; the cache hit is automatic.

If the model download itself fails you’ll see:

failed to download onnx/model.onnx from <repo> (check network or set duckdb.model_cache to a pre-populated dir)

Q: The first run fails on tokenizer.json with “relative URL without a base”. What’s wrong?

You are running a build older than the hf-hub 0.5 upgrade. The old pinned hf-hub 0.3 client could not fetch files that Hugging Face stores in Xet storage — for the recommended code model jinaai/jina-embeddings-v2-base-code, onnx/model.onnx downloaded fine but tokenizer.json errored with relative URL without a base.

Fix: upgrade to a current build; the first-run download of both files now works. If you must stay on an old build, stage tokenizer.json into the model’s HF cache snapshot once with curl, then run normally:

# Old builds only: stage tokenizer.json into the cached snapshot for the jina code model
SNAP=~/.cache/huggingface/hub/models--jinaai--jina-embeddings-v2-base-code/snapshots/*/
curl -sL https://huggingface.co/jinaai/jina-embeddings-v2-base-code/resolve/main/tokenizer.json -o $SNAP/tokenizer.json

See Choosing a model and Backends and embedders for the full model recommendation.

Q: I get “embedder produced N-d vectors but vector_dim=M”. How do I fix it?

The DuckDB table’s embedding column is declared FLOAT[vector_dim], so the embedder’s output dimension MUST equal the configured vector_dim. A mismatch means the model and vector_dim disagree, or you switched models without rebuilding the index. The runtime guard prints the right dimensions for the common models:

embedder produced 768-d vectors but vector_dim=384 — set vector_dim to match the model (e5-small=384, nomic-embed-text=768, mxbai-embed-large=1024)

When you open an existing index whose stored column type no longer matches, you get the column-level variant instead:

DuckDB table '<collection>' has embedding column of type FLOAT[384] but config/vector_dim=768 (expected FLOAT[768]). This usually means the embedding model was changed without --recreate. Delete the DuckDB file or re-index with --recreate.

Fixes (any one):

1. Set vector_dim to match your model — e5-small = 384, nomic-embed-text = 768, mxbai-embed-large = 1024, jinaai/jina-embeddings-v2-base-code = 768.

2. Re-index with --recreate (drops and recreates the collection):

   ./target/release/semanticastindexer --backend duckdb --recreate --root src --ext rs
   

3. Delete the DuckDB file (and its .wal sidecar) and re-index:

   rm -f .index/code.duckdb .index/code.duckdb.wal
   

The interactive delete prompt. On a normal index run (not --query-only), if SAI opens an index that has a dimension mismatch, it offers to delete and rebuild for you:

The index at '<path>' was built with a different embedding model (dimension mismatch). Delete it and re-index from scratch? [y/N]

This prompt defaults to No and only deletes on an explicit y/yes. When stdin is not an interactive terminal (CI, git hooks, the MCP stdio server), it auto-declines and the original error propagates — automation never blocks on input and never destroys an index by default. A --query-only run never re-indexes, so it just surfaces the error rather than offering to delete (deleting would only leave an empty DB to query).

Q: I get “Document API only works on Qdrant Cloud”. Why?

The qdrant backend uses Qdrant’s server-side inference (the Document API) — there is no local model on this path; the cluster embeds your text. Plain OSS/local Qdrant has no inference engine, so the Document API only works against Qdrant Cloud (or another inference-enabled deployment) with Inference enabled and the embedding model available on the cluster.

Fixes:

• Use a Qdrant Cloud cluster with Inference enabled and the embedding model present (intfloat/multilingual-e5-small, vector size 384, context window 512). Provide the API key in the environment (the URL can be qdrant.url in YAML or QDRANT_URL):

  export QDRANT_URL="https://<cluster-id>.<region>.aws.cloud.qdrant.io:6334"   # gRPC :6334
  export QDRANT_API_KEY="<key from the cluster's API Keys tab>"
  

• Or switch to the fully local DuckDB backend, which embeds on-device and needs no cluster:

  ./target/release/semanticastindexer --backend duckdb --root src --ext rs
  

See Qdrant Cloud setup and the Environment reference.

Q: I get a “rebuild with –features …” error. What does that mean?

Backends and embedders are compiled behind Cargo feature flags. Selecting a backend or embedder whose feature was not built into the binary fails with a clear “rebuild with –features …” message. Likewise, if no embedder feature is compiled in, the embedder errors with:

no embedder compiled in (build with --features ort or --features ollama)

Fix: rebuild with the feature you need. The simplest catch-all enables everything:

cargo build --release --features all

Or enable just what you use, for example the local DuckDB + ONNX + AST path:

cargo build --release --features "duckdb,ort,ast"

The ast chunker is also feature-gated — selecting chunker: ast (or auto-selecting it) without the ast feature errors early. See Backends and embedders for the feature matrix.

Q: I get zero results, or duplicates/similar misses obvious matches. Why?

A few independent causes:

• The index is empty. Run an index first. With the DuckDB MCP server, a missing index is an explicit error: DuckDB index not found at <path> — run an index first (the MCP server is read-only).

• Recall degraded after deletes (DuckDB only). DuckDB’s experimental HNSW index loses recall after in-place deletes. SAI’s sync therefore wraps its changed-file loop in a bulk window (begin_bulk / end_bulk) that drops and recreates the HNSW graph, restoring full recall — effectively a full rebuild of the graph. If rows were deleted by some other means, run a sync (or re-index) so the graph is rebuilt:

  ./target/release/semanticastindexer sync --backend duckdb
  

• Threshold too high for your model. Code-trained models score lower than E5, so a duplicate_min_score tuned for E5 can hide real near-duplicates. Lower the threshold for the jina code model (e.g. 0.88). See Tuning similarity.

• Wrong prefix style for the model. Both embedders apply E5’s passage:/query: prefixes by default; a symmetric or non-E5 model wants prefix_style: none, otherwise relevance suffers.

Q: My MCP client doesn’t show the sai_* tools. How do I wire it up?

A handful of wiring issues account for almost every “the server doesn’t appear” report:

• Use an absolute command path. Point the client at the built binary by absolute path, e.g. /abs/path/to/target/release/semanticastindexer, not a bare name your client may not resolve.
• Set the working directory to the indexed project. The server resolves the index and config relative to its cwd, so the client must launch it with cwd = the project you indexed (where .index/code.duckdb / sai-cfg.yml live).
• Restart the client. Most MCP clients only read server config at startup; after editing the config, fully restart the client so it spawns the server again.
• The server is read-only by default. It runs with --backend duckdb --embedder ollama defaults and is read-only; the sai_refresh write tool is not registered unless you start it with --allow-write. Without that flag, sai_refresh returns a clear “restart with –allow-write” error. The sai_prepare_mcp_setup tool only executes the setup script when started with --allow-setup.
• Index first. A read-only server cannot index; if .index/code.duckdb does not exist yet, run an index before launching the server.

Minimal stdio launch (read-only) and the writable variant:

# read-only MCP server over stdio (defaults: duckdb + ollama)
/abs/path/to/target/release/semanticastindexer mcp

# writable: also registers the sai_refresh tool
/abs/path/to/target/release/semanticastindexer mcp --allow-write

The full list of sai_* tools and their argument schemas lives in the MCP server reference; per-client config snippets are in MCP clients.

Q: The MCP server can’t load the DuckDB VSS extension. What now?

The DuckDB backend needs the VSS extension (for HNSW search and array_cosine_distance). SAI tries a plain LOAD vss first (works on read-only connections if VSS was previously installed), then INSTALL vss; LOAD vss, then the community repo. If all fail you get an actionable error. The common fix is to install VSS once with a writable run:

• Run the indexer at least once with write access so it can INSTALL vss, or
• Pre-install for a read-only MCP server: duckdb -c "INSTALL vss;" as a user who can write to DuckDB’s extension directory, or
• In an air-gapped environment, copy the VSS extension into DuckDB’s extension search path before starting the read-only server.

See Performance and Security for related operational notes, and Keeping in sync for the sync workflow that keeps recall healthy after edits.

Performance and scaling

This page explains where SAI spends time and memory, and how to keep an index fast on large repositories. It is honest about what is measured and what is not: no benchmarks ship with SAI. The numbers below are architectural constants read from the source (batch sizes, thread counts, token windows), not throughput figures. Treat the guidance as “where the costs are,” then measure on your own repo.

Where the time goes

For the local ort (ONNX) embedder, indexing is dominated by two costs:

1. CPU embedding — running the ONNX model forward over every chunk.
2. HNSW index rebuild — dropping and recreating the DuckDB vector index around each bulk write.

For the Qdrant backend, embedding happens server-side (Inference), so client-side cost is mostly network round-trips for upsert and search.

ort embedding: CPU inference in batches of 32

The ort embedder runs synchronous CPU inference. Two constants set its shape:

• Batch size 32. Passages are embedded 32 per ONNX forward pass (EMBED_BATCH).
• All cores. The ONNX intra-op thread pool is sized to the machine: intra_threads = std::thread::available_parallelism() (falling back to 1 if the count is unavailable). Indexing is a throughput-bound, one-shot batch job, so every core works the forward pass.

To reduce wasted compute on padding, each batch is length-sorted before tokenization (so a single long passage does not inflate its whole batch under BatchLongest padding), then results are scattered back to the caller’s order. Tokens are truncated/padded to the 512-token E5 context window (MAX_TOKENS).

Because inference is CPU-bound and uses all cores, the practical levers are: fewer/smaller chunks (see max_chunk_chars below), a faster model (see model trade-offs), or offloading to a server (Qdrant Inference or Ollama).

The HNSW drop-and-rebuild cost (DuckDB)

DuckDB’s HNSW vector index makes per-row INSERT expensive (the graph is maintained per row). To avoid that, every bulk write drops the HNSW index, inserts, then recreates it:

• begin_bulk() runs DROP INDEX IF EXISTS <collection>_hnsw.
• All upserts happen with no index present (each batch runs as one transaction).
• end_bulk() runs CREATE INDEX ... USING HNSW(embedding) WITH (metric='cosine'), rebuilding the whole graph from scratch.

This rebuild is a real cost on large repos: the index is rebuilt over every stored vector, not just the ones you changed. A full index pays it once. But every sync also wraps its deletes + upserts in begin_bulk/end_bulk — so even a sync that touches one file rebuilds the entire HNSW index at the end. On a large collection that rebuild can dominate a small sync.

Two consequences:

• DELETE is cheap. Deleting a path’s rows does not trigger an HNSW rebuild, so delete_by_path needs no index teardown. The cost is the recreate in end_bulk.
• Sync cost scales with index size, not change size. If syncs feel slow on a huge repo, the rebuild is the likely cause. Sync less often, or scope the index smaller (see large monorepos). See Keeping in sync for when syncs run.

HNSW persistence on a file-backed DuckDB database is experimental and is enabled with SET hnsw_enable_experimental_persistence = true. SAI sets this automatically on the writable connection; the read-only MCP server does not enable it (it never writes).

Memory footprint of the ONNX session

The ort embedder owns an ort::Session (the loaded ONNX model graph + weights) plus a tokenizer, held for the life of the process. That resident footprint scales with the model:

• e5-small (384-dim) is small and light.
• jina code (768-dim) and Ollama-served large models (e.g. 1024-dim) are larger.

Per-batch working memory is bounded by the batch (32 passages) times the padded sequence length (up to 512 tokens) times the hidden dimension — modest next to the resident weights. The single largest lever on memory is therefore model choice, not batch size. The Embedder enum boxes the Ort variant precisely because the ONNX session is far larger than the Ollama (HTTP-only) variant.

max_chunk_chars: relevance vs cost

max_chunk_chars caps the size (in characters) of a single chunk. It is model-aware by default:

• E5 / Qdrant path: 1400 chars (E5’s 512-token window ≈ 1400 chars).
• Large-context / code models (jina, Qwen, Ollama large models on the DuckDB path): a much larger cap, so a whole function fits in one chunk.

The trade-off:

• Smaller chunks → more chunks → more embedding passes and more rows. Each chunk is more focused, which can sharpen search relevance, but you pay more compute and storage, and a function may be split across windows (the line chunker overlaps windows by 8 lines to soften this).
• Larger chunks → fewer, broader vectors → cheaper to index, but a single vector now averages over more code, which can blur near-duplicate detection and dilute search precision.

You can override the default explicitly:

# sai.toml
max_chunk_chars = 2000

Keep the cap consistent with the model’s real context window — a 1400-char cap on an 8K-token model wastes capacity, and a huge cap on a 512-token E5 model just gets truncated at tokenization. See Chunking for how chunks are formed and Configuration for the key.

Model speed vs quality

There is a direct speed/quality/footprint trade-off across the three embedding paths:

Reading:

• e5-small is the small/fast choice — lowest CPU cost and memory, general-purpose, used as the default for Qdrant server inference and as a lightweight ort option.
• jina code is the quality choice for code on the local ort path — better code understanding at a larger, slower model.
• Ollama moves embedding off the indexing process to a separate server; throughput then depends on that server’s hardware, and the model is required (no built-in default — set ollama.model).

Whichever you pick, vector_dim must match the model (e5-small=384, jina/nomic=768, mxbai-embed-large=1024) or the index rejects the dimension. Changing models requires --recreate (or deleting the DuckDB file). See Choosing a model and Backends and embedders.

Qdrant upsert batch size

On the Qdrant backend, points are upserted in batches of 32 (UPSERT_BATCH = 32), kept modest because server-side inference runs per request. Each batch is sent with wait(true). (Separately, the CLI’s embed+upsert loop groups chunks 64 at a time before handing each batch to the backend; for Qdrant those are then re-chunked to 32 per request.)

If Qdrant indexing feels slow, the bottleneck is typically the cluster’s Inference throughput and network latency, not the client.

Tips for large monorepos

The cheapest way to stay fast on a huge repo is to index less. Scope tightly:

• Narrow the root. Point --root at the subtree you actually search instead of the repo root:

  sai index --root services/api
  

• Restrict extensions. Limit --ext to the languages you care about so the walk skips everything else:

  sai index --root services/api --ext rs,ts
  

• Use include globs. Include only the paths worth indexing; everything else is excluded before it is ever read or embedded:

  # sai.toml
  include = ["src/**", "lib/**"]
  exclude = ["**/*.generated.*"]
  

• Index subtrees into one collection. Run sai index over several subtrees in turn, all targeting the same collection, to build one searchable index from selected parts of a large tree without indexing the whole thing. Use dry-run first to see exactly what would be indexed and why files are excluded.

These reduce both costs at once: fewer chunks means fewer embedding passes, and a smaller collection means a cheaper HNSW rebuild on every sync.

Other practical measures:

• Sync deliberately. Because each sync rebuilds the whole HNSW index, batch your changes and sync once rather than after every edit. See Keeping in sync.
• Skip generated and opt-out code. Generated files and sai-noindexing spans are dropped before embedding, which keeps the index focused and smaller. See Opt-out markers.
• Reuse the model cache. Set the model cache directory so the model is downloaded once and reused offline across runs and CI. See CI/CD and Environment.

What is not measured

SAI ships no benchmark suite and no published throughput numbers. The constants here — batch 32, 512-token window, all-core intra-op threads, drop-and-rebuild HNSW, Qdrant batch 32 — are real and load-bearing, but actual wall-clock time depends entirely on your hardware, model, repo size, and chunk cap. Measure on your own repo, change one variable at a time (model, max_chunk_chars, scope), and compare.

If indexing or search is slower than expected, see Troubleshooting.

Security and privacy

This page consolidates everything about credentials, what data leaves your machine, and the capabilities the MCP server can expose. Two things drive the security model: where credentials come from and which backend you choose.

The API key is a secret; the URL is not

The only secret SAI uses is the Qdrant API key, and it is read only from the environment — never from sai-cfg.yml or any other config file. The cluster URL is not a secret and may live in YAML (qdrant.url) or in the environment.

Rules:

• Never commit the API key. Keep it out of YAML, .mcp.json, and shell history that lands in version control. The URL is safe to commit.
• Rotate any exposed key. If the API key is ever leaked, rotate it in the cluster’s API Keys tab and update QDRANT_API_KEY.
• These only matter for the Qdrant backend. The local DuckDB backend needs no credentials at all.

# The key always comes from the environment; the URL can come from YAML or here.
export QDRANT_API_KEY="<your-key>"
export QDRANT_URL="https://<your-cluster>.qdrant.io:6334"   # or set qdrant.url in sai-cfg.yml
semanticastindexer --backend qdrant --root src --ext ts,tsx

See environment variables for the full list and Qdrant Cloud for cluster setup.

What leaves the machine

The biggest privacy lever is the backend. With the local backend, your source code never leaves your machine; with Qdrant Cloud, your code text is sent to the cluster for server-side inference and storage.

Choose duckdb + ort for a fully offline, nothing-leaves-the-machine setup; use duckdb + ollama against your own local server for the same privacy with a separate embedding process; and treat qdrant as a deliberate decision to send code to a third-party cloud service.

MCP server capabilities

The MCP server (semanticastindexer mcp) is read-only by default. Read-only tools embed queries and run nearest-neighbour searches against the existing index — they never mutate it. Two flags unlock additional capabilities; both are off by default.

Read-only by default

With no extra flags, the server exposes only read tools:

• sai_search_code — semantic search over the index.
• sai_find_similar — neighbours of a snippet or an existing chunk.
• sai_find_duplicates — codebase-wide near-duplicate clusters.
• sai_index_status — backend / collection / model / dimension / chunk count / chunker.

The setup-helper tool sai_prepare_mcp_setup is also present but, without --allow-setup, only returns the commands and config you should run — it does not execute anything.

--allow-write — enables the write tool

--allow-write enables sai_refresh, which mutates the index: for each supplied path it deletes existing points, then re-chunks, re-embeds, and re-upserts the files that still exist and pass the index filters (a single call is capped at 200 paths). Without this flag, sai_refresh returns a clear error and the backend is opened read-only.

--allow-setup — lets the server execute the setup script

--allow-setup is a meaningful capability: it lets sai_prepare_mcp_setup actually execute the setup script when a caller passes execute: true. The setup script can build the binary and modify files (it runs bash -c with the resolved setup command in the target directory). Without --allow-setup, an execute: true request is blocked and the tool only reports the commands.

Treat --allow-setup as you would any tool that can run builds and change files on disk: leave it off unless you specifically want an agent to drive setup.

# Default: read-only, no execution.
semanticastindexer mcp --backend duckdb --embedder ort

# Allow the agent to re-index files in place:
semanticastindexer mcp --backend duckdb --embedder ort --allow-write

# Additionally allow the agent to execute the setup script (build + file changes):
semanticastindexer mcp --backend duckdb --embedder ort --allow-write --allow-setup

The full tool list, schemas, and threshold defaults are documented in the MCP server reference.

Safety note: the sai-noindexing marker

The literal string sai-noindexing is an opt-out marker: a file (or chunk) containing it is dropped from the index. Because this is a plain substring match, the marker can also match inside a string literal in your source — for example a constant whose value is "sai-noindexing". When that happens, the surrounding code is silently dropped from the index even though you did not intend to exclude it.

If a file you expect to find never shows up in search results, check whether the sai-noindexing string appears anywhere in it, including inside string literals.

.gitignore

Add target/ to your .gitignore — it is a build artifact and should never be committed.

target/

Contributing

Thanks for hacking on semanticastindexer (SAI). This page is the developer setup guide: how to build and test the crate, what the Cargo feature flags gate, how the src/ tree is laid out, and how to run the MCP server locally for debugging. Before changing anything load-bearing, read how it works for the indexing pipeline and the invariants it must preserve.

Toolchain

• MSRV 1.88, edition 2024 (declared in Cargo.toml as rust-version = "1.88" / edition = "2024"). The floor is imposed by ort 2.0.0-rc.12.
• The repo ships a rust-toolchain.toml that pins the stable channel and installs the rustfmt and clippy components. rustup activates it automatically when you build inside the repo.

# rust-toolchain.toml
[toolchain]
channel = "stable"
components = ["rustfmt", "clippy"]
profile = "minimal"

Build & test

Always build with --features all — the canonical, supported configuration. It gives you a single binary with every backend, embedder, the AST chunker, and the MCP server.

cargo build --release --features all   # full-featured binary
cargo test  --release --features all   # run the test suite

The first --features all build is slow (it compiles bundled DuckDB and ONNX Runtime); subsequent builds are incremental and fast.

The Makefile wraps the same commands and is the easiest way to drive the binary. Key targets (run from the repo root):

make clippy and make check-all treat warnings as errors (-D warnings), so run one of them before opening a PR.

The build-ort, build-ollama, and build-ast targets are legacy aliases that all just call make build (the full --features all build).

Cargo feature matrix

Features are declared in [features] in Cargo.toml. The two you care about are default (a minimal Qdrant-only build) and all (everything).

Notes drawn straight from the manifest:

• The DuckDB backend has no embedder of its own; pair duckdb with ort or ollama. Both embedder features imply duckdb.
• mcp defaults to backend=duckdb + embedder=ollama, which is why it pulls in those features so the server is usable out of the box.
• twox-hash is non-optional: point_id uses XxHash64 (seed 0) unconditionally so Qdrant and DuckDB produce identical, stable point IDs.
• The release profile uses opt-level = 3 + lto = "thin" (the embedder’s mean-pool / L2-normalize loops and the tokenizer are hot, CPU-bound code).

Module map (src/)

src/
├── main.rs              # CLI entry point (clap): parses args, dispatches subcommands
├── config.rs            # sai-cfg.yml parsing, filters, sai-* opt-out markers
├── git.rs               # git helpers (changed-files / since for `sync`)
├── indexer.rs           # indexing pipeline: chunk → embed → upsert
├── search.rs            # query / similar / duplicates logic
├── worker.rs            # actor thread that owns the !Send DuckDB/ort resources
├── mcp.rs               # MCP server (rmcp #[tool] handlers, read-only)
└── vectordbs/
    ├── mod.rs           # Backend enum + shared types (Hit, etc.)
    ├── qdrant.rs        # Qdrant backend (feature = "qdrant")
    ├── duckdb.rs        # DuckDB backend (feature = "duckdb")
    ├── embedder.rs      # ort / ollama embedders
    └── mock.rs          # in-memory test backend (#[cfg(test)] only)

worker.rs exists because rmcp’s tool-handler futures must be Send, but the DuckDB Connection (and the ort ONNX session) are !Send/!Sync. The worker moves those resources onto a dedicated OS thread running its own current-thread Tokio runtime; the MCP server holds only a Send channel handle. Keep that boundary intact when touching the MCP path.

The mock backend (tests, no network)

src/vectordbs/mock.rs is an in-memory backend compiled only under #[cfg(test)] — it never ships in a release binary. It runs the real orchestration code (index_sources, sync, run_query, flush) with no network and no real Qdrant/DuckDB, and records every backend call so tests can assert ordering, balance, and arguments. It also seeds rows-with-vectors so the MCP-path methods (query_by_vector, get_by_location, all_chunks_with_vectors) can be tested without a real backend. This is why cargo test --features all needs no credentials or running services.

Running the MCP server locally

The MCP server is gated behind the mcp feature and speaks the protocol over stdio. To run it against a built binary:

cargo build --release --features all
./target/release/semanticastindexer mcp

The server is read-only by default; the write tool requires --allow-write. For the full list of shipped tools (sai_search, sai_similar, sai_duplicates, and the rest), their input schemas, and how to wire the server into a client’s .mcp.json, see the MCP server reference. The mcp build defaults to backend=duckdb + embedder=ollama, so for an end-to-end debug session index a project into DuckDB first (e.g. make prod TARGET=. BACKEND=duckdb) before launching the server over it.

Pull requests

• Use Conventional Commits for commit messages and PR titles (feat:, fix:, refactor:, docs:, test:, chore:, perf:, ci:).
• Run make fmt and make check-all (clippy with -D warnings) before opening a PR — CI builds and tests with --features all.
• Add or update tests using the mock backend; they must pass offline.
• Respect the core invariants (see how it works → Invariants): stable point IDs, the !Send worker boundary, the DuckDB begin_bulk/end_bulk contract, and the read-only-by-default MCP server.

Changelog

All notable changes to SemanticAstIndexer (SAI) are recorded in the project’s CHANGELOG.md. The changelog format follows Keep a Changelog, and the project adheres to Semantic Versioning.

For downloadable builds and per-version release notes, see the GitHub releases page.

The entries below are reproduced from the in-repo changelog; the file in the repository is always the source of truth.

[0.1.0] - 2026-05-31

Initial release.

Added

• Semantic AST code indexer with pluggable vector backends: Qdrant (Cloud server-side inference) and DuckDB (local VSS/HNSW cosine index).
• Pluggable embedders for the DuckDB backend: ort (local ONNX Runtime, offline) and ollama (remote HTTP).
• Model-aware embedding prefixes (E5 / Qwen / none).
• Pluggable chunker: line-window (default) and AST (tree-sitter, symbol-aware) for TypeScript/TSX, Rust, and Go.
• YAML configuration controlling excluded dirs/globs, generated-marker skip, and comment stripping.
• CLI commands: index, sync, flush, --dry-run, --query/--query-only, similar, duplicates.
• MCP server (mcp subcommand) exposing read-only semantic search tools over stdio.
• Cargo feature matrix: qdrant (default), duckdb, ort, ollama, ast, mcp, all.

Versioning & compatibility

SAI follows Semantic Versioning. While the project is in the 0.x series, treat minor releases as potentially breaking per the SemVer pre-1.0 rule, and read the release notes on the GitHub releases page before upgrading.

Index-format compatibility

The index is produced by a specific combination of backend, embedder, model, and chunker. Two indexes are only directly comparable when those are the same:

• Embedding model. A collection embedded with one model cannot be mixed with vectors from another — the vector spaces differ. Searching across a model change requires a re-index.
• Backend. Qdrant and DuckDB store independent indexes; switching backends means building a fresh index for that backend.
• Chunker. Switching between the lines and ast chunker changes how source is split and how point IDs are derived, so existing points should be rebuilt.

When in doubt after changing any of these, recreate the collection (flush, or index with --recreate) so stale points from the previous configuration do not linger. See ../reference/cli.md for the full flag reference.

One-time re-index migration (point-ID hashing)

Point IDs are a stable XxHash64(seed=0) of path + start_line. This replaced the earlier, unspecified DefaultHasher. Because the ID computation changed, existing collections must be flushed or recreated once so that old points (keyed under the previous hashing scheme) do not linger alongside the new ones:

# Option A: delete the whole collection, then index fresh.
./target/release/semanticastindexer flush
./target/release/semanticastindexer --root src --ext ts,tsx --collection source_code

# Option B: drop & recreate the collection in a single indexing run.
./target/release/semanticastindexer --root src --ext ts,tsx \
    --collection source_code --recreate

After the re-index, point IDs are stable across subsequent sync runs, so incremental updates correctly replace the points for changed files. See ../reference/cli.md for flush, sync, and --recreate.