Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Indexing a project

Building an index is how SAI turns your source tree into searchable vectors. This guide walks through the full task: choosing what to walk, previewing the selection, running the real index, and reading the result. Run the binary from the target project’s repo root so the stored paths are project-relative (or point --root at the project’s source dir).

For the keys referenced here, see Configuration. For how a file becomes chunks, see Chunking. To exempt code from indexing, see Opt-out markers.

What to walk: --root and --ext

Two flags decide which files are even considered:

  • --root <dir> — the directory to walk. Default: src.
  • --ext <list> — comma-separated extensions, no dots. Default: ts,tsx.
BIN="$(pwd)/target/release/semanticastindexer"   # absolute path to the built binary

# Move into the project you want to index (so payload paths are project-relative).
cd /path/to/your/project

# Index the TypeScript tree.
"$BIN" --root src --ext ts,tsx --collection source_code

Only files whose extension appears in --ext are read. Both flags override sai-cfg.yml.

Dry-run first

Always preview the selection before a real index. --dry-run walks the tree and reports exactly which files would be indexed and which are excluded (and why) — no network, no upload, no quota used:

"$BIN" --root src --ext ts,tsx --dry-run

The report prints the resolved root/ext/collection/model, the active strip_comments / skip_generated_marker settings, the pruned directory names, a WOULD INDEX / EXCLUDED count, a per-reason breakdown (glob, not-included, generated-marker), and a sample of included and excluded paths. The dry-run uses the same shared decision function as the real index, so what it reports is what you get.

Re-indexing: --recreate

By default, indexing creates the collection/table if missing and upserts chunks in place (re-running updates existing points, because each point ID is a stable hash of path + start_line). Pass --recreate to drop and recreate the collection before indexing — a clean slate:

"$BIN" --root src --ext ts,tsx --recreate

A one-time re-index is required for collections built before point IDs became a stable XxHash64(seed=0) of path + start_line. Run "$BIN" flush or index once with --recreate so stale points don’t linger.

Selection order

For each file under --root that survives directory pruning and the --ext filter, the include/exclude decision runs in this exact order:

  1. include allow-list — if include is non-empty, the file must match one of its globs, otherwise it is skipped (reported as not-included).
  2. exclude globs — if the path matches an exclude glob, it is skipped. Exclude always wins over include.
  3. Hard-pruned directories — certain directory names are pruned during the walk regardless of config: node_modules, .git, dist, build, target, .next, coverage, .turbo. Names in exclude_dirs are pruned too.
  4. skip_generated_marker — when enabled (default true), the first ~600 bytes of the surviving file are scanned for autogenerated markers (@generated, DO NOT EDIT, code generated, auto-generated, autogenerated, this file is generated). A match skips the file, catching generated files that don’t follow a naming convention.
  5. strip_comments — when enabled (default true), C-family // and /* */ comments are removed before embedding so only code reaches the backend. String/template literals are preserved and line numbers stay accurate.

Steps 1–2 are the glob gate; the hard-pruned dirs (step 3) are applied as the walk descends, before any file is even examined.

A minimal sai-cfg.yml controlling these:

include: []                       # empty → consider all files; non-empty → allow-list only
exclude:
  - "**/*.test.ts"
  - "**/*.d.ts"
  - "**/components/ui/**"          # shadcn primitives
  - "**/*.pb.go"                   # Go autogenerated
exclude_dirs:
  - __tests__                      # extra dir names to prune (beyond the hard-coded set)
skip_generated_marker: true
strip_comments: true

Per-extension language labels

Each chunk is stamped with a language payload label derived per file from its extension, lowercased: .ts → ts, .tsx → tsx, Bar.TSX → tsx. So a single --ext ts,tsx walk labels each file with its own language, and you can later filter search or duplicate scans by that label.

Reading the result

A successful index ends with one summary line on stdout:

indexed 1843 chunks from 211 ts/tsx file(s) into 'source_code' (37 file(s) skipped by config)

Reading it left to right: the chunk count, the file count, the extensions (--ext joined by /), the target collection, and (N file(s) skipped by config) — files dropped by the glob gate or the generated-marker scan. (Binary / non-UTF-8 files are silently ignored and are not counted as skipped.) During embedding, progress is printed to stderr; only this final line goes to stdout.

Indexing more languages into the same collection

You can index additional trees into an existing collection by re-running with a different --root / --ext. New points are added; existing ones are updated in place:

"$BIN" --root path/to/go --ext go --collection source_code

Next steps