How it works

SAI (semanticastindexer) turns a codebase into a vector index of code chunks, then answers semantic queries and near-duplicate scans against that index. This page traces the two data paths end to end — indexing (source files → stored vectors) and search (a query → ranked hits) — and explains the worker-thread actor model the MCP server uses to drive a !Send backend safely.

For the user-facing surface, see the CLI reference and the MCP server and tools page. Terms in bold are defined in the glossary.

The indexing pipeline

Indexing walks the project root and transforms each surviving file into embeddable chunks:

walk --root → prune dirs → ext filter → include allow-list → exclude globs
            → generated-marker skip → strip comments
            → chunk (lines: ~60 lines / max_chunk_chars, 8-line overlap; or ast: per symbol)
            → Document("passage: <code>", model) → upsert (batch, server-side embed)

The pure chunking paths (collect_chunks, build_chunks, dry_run) never touch a vector backend — they only decide what to embed. A single bridge function, reindex_file, performs storage I/O (delete + upsert) for one file, and it is shared by both CLI sync and the MCP sai_refresh tool.

Per-file flow in detail

For each file under --root (after directory pruning and the extension filter), the single source-of-truth decision function load_file_for_indexing applies, in order:

1. include (allow-list) — if the include set is active, the file must match an include glob, else it is skipped.
2. exclude globs — if matched, the file is skipped (exclude always wins over include). Include and exclude are evaluated together by passes_globs.
3. skip_generated — the first 600 bytes are scanned (case-insensitively) for an autogenerated marker (do not edit, @generated, code generated, auto-generated, autogenerated, this file is generated); matches are skipped.
4. The surviving content is read as UTF-8. Binary / non-UTF-8 files are silently ignored.

Then build_chunks produces the embeddable chunks:

1. strip_comments (for C-family extensions) removes // line and /* */ block comments before chunking, so only code is embedded. The stripper preserves string, char, and template literals — and preserves the exact line count, so a chunk’s start_line/end_line still point at the real lines in the original file.
2. The surviving content is split into chunks (see chunking).
3. Each chunk’s raw (pre-strip) line span is scanned for opt-out markers (sai-noindexing drops the chunk entirely; sai-noduplicate flags it so it is stored but excluded from clustering). See opt-out markers.
4. Each surviving chunk is embedded and upserted into the configured backend.

C-family extensions covered by comment stripping are: ts, tsx, js, jsx, mjs, cjs, go, rs, java, c, h, cc, cpp, hpp, cs, kt, swift, scala.

Chunkers

The AST index is deliberately function-only: non-function code (imports, classes, interfaces, type aliases, plain consts, bare closures) is never emitted, so a file with no functions produces no chunks. Selecting chunker: ast on a binary built without the ast feature is a hard startup error. Full details live in chunking.

Deterministic point IDs

Every chunk’s storage ID is a deterministic hash of path + start_line:

#![allow(unused)]
fn main() {
pub fn point_id(path: &str, start_line: usize) -> u64 {
    let mut hasher = XxHash64::with_seed(0);
    path.hash(&mut hasher);
    start_line.hash(&mut hasher);
    hasher.finish()
}
}

XxHash64 with a fixed seed = 0 is stable across builds and machines (unlike DefaultHasher, whose algorithm is unspecified). Because the ID is a pure function of path + start_line, re-running the indexer updates points in place instead of duplicating them, and every backend keys the same chunk identically — so Qdrant and DuckDB agree on IDs.

Payload shape

Each stored point carries this payload:

Chunks also carry a no_duplicate flag (set by the sai-noduplicate marker) and git context (commit_sha, dirty) stamped at index/refresh time.

The search path

All semantic queries embed the input, run a nearest-neighbour (NN) search, then post-filter the over-fetched results. Three shaped reads back the shipped MCP tools and their CLI twins:

• Query search (sai_search_code) — embed text as a query vector, NN-search, return hits. The backend over-fetches (fetch) so a score cut can be applied afterwards. When the backend cannot embed locally (Qdrant), the worker falls back to the backend’s server-side text query() path.
• Similar search (sai_find_similar) — resolve the target, NN-search, then drop everything below min_score:
  • A code snippet is embedded as a passage (code-vs-code space) and searched by that vector.
  • A location (path + 1-based line) reuses the stored chunk’s exact vector — no re-embed — and excludes the chunk itself from its own results.
• Duplicates scan (sai_find_duplicates) — fetch every stored chunk (optionally path-glob filtered), gather each chunk’s top_k nearest neighbours (self-excluded, stored vectors — no re-embed), then run the pure clustering core.

Near-duplicate clustering

cluster_duplicates is a pure union-find over per-chunk neighbour lists, shared by the CLI and the MCP server so the algorithm exists in exactly one place. An edge is kept when its similarity >= min_score; kept edges union the two chunks. Chunks flagged no_duplicate form no edges — neither as a seed nor as a neighbour. Connected components with >= min_cluster_size members become clusters, sorted largest-first (tie-break: higher max_sim) and truncated to max_clusters. Each cluster reports its members plus the min/max edge similarity within it. See tuning similarity for choosing thresholds, and search and duplicates for usage.

The backend worker thread (actor model)

The DuckDB backend embeds a duckdb::Connection (and, with the ort embedder, an ONNX session) that are !Send/!Sync. But the MCP framework (rmcp) requires every tool handler future to be Send, so holding such a backend across an .await inside a tool handler would not compile under the mcp feature.

SAI solves this with an actor / worker-thread pattern:

  MCP server (multi-thread runtime)            backend worker (one OS thread)
  ┌───────────────────────────┐                ┌─────────────────────────────┐
  │ #[tool] handler            │  Request +     │ current-thread Tokio runtime │
  │   BackendHandle (Send+Sync)│  oneshot reply │   owns the !Send Backend     │
  │     mpsc::Sender ──────────┼───────────────▶│   worker_loop: one at a time │
  │   .await oneshot ◀─────────┼────────────────┤   handles each fully, replies│
  └───────────────────────────┘                └─────────────────────────────┘

• A dedicated OS thread (semanticastindexer-backend) owns the Backend and the Plan. It runs its own current-thread Tokio runtime so it can drive the backend’s async methods (e.g. the Ollama embedder’s reqwest calls) locally. The !Send backend never crosses a thread boundary.
• The MCP server holds only a BackendHandle — a Send + Sync wrapper around an mpsc::Sender<Request>. Channels are Send + Sync, so each tool handler captures only Send types and compiles.
• Every tool call builds a Request plus a oneshot reply channel, sends it, and .awaits the reply. The worker loop processes requests one at a time — each request is fully handled (including its .awaits) before the next is taken — so the single DuckDB connection is never touched concurrently. The request channel is bounded (buffer of 32), which is plenty for a strictly sequential consumer.
• The pattern is backend-agnostic: Qdrant’s backend is already Send, but routing every backend through the same worker keeps the handler code identical.

Dropping every BackendHandle clone closes the channel, which ends the worker loop and the thread.

How sync and refresh bracket HNSW bulk windows

DuckDB’s vector index (HNSW) requires bulk mutations to be bracketed so the index is dropped before mass upserts and rebuilt afterwards. Both write paths honour this contract:

• The shared per-file step reindex_file always deletes the file’s existing points first, then — if the on-disk path is still indexable — re-chunks, embeds, and upserts the fresh chunks. The caller is responsible for wrapping the whole operation in a single begin_bulk() / end_bulk() window.
• The MCP sai_refresh tool’s handle_refresh mirrors sync’s correctness contract: begin_bulk (drop HNSW) → per-path delete + re-chunk + re-embed + upsert → end_bulk (rebuild HNSW). A fresh git context is captured per path so the commit_sha/dirty stamp is accurate on a long-lived server.

There is a logical invariant here: end_bulk is always called, even when a path fails mid-batch, so the HNSW index is never left dropped after a refresh. sai_refresh is the single write path exposed by the MCP server and is gated by --allow-write at the call site. See keeping the index in sync and the MCP server reference.

Invariants

A few algorithmic invariants are load-bearing and enforced in the code itself: chunking “nothing dropped” (the dry-run, full-index, and refresh paths share one per-file routine), the DuckDB HNSW bulk contract (begin_bulk/end_bulk around every write), stable cross-backend point IDs, one resolved prefix style applied everywhere, the !Send worker boundary, and the runtime dimension guards. Re-read the relevant modules (src/indexer.rs, src/vectordbs/, src/worker.rs, src/search.rs) before changing core indexing, clustering, or backend logic.