perf(index): parallelize per-file read, hash, chunk, and tokenize (#115)

[amondnet]

Summary

This ports upstream semble #295, which parallelizes the per-file indexing work in create_index_from_path.

• Phase 1 runs in parallel on a local rayon pool, in batches of 256 files to bound memory. For each file it checks size, reads, hashes, and, when the file is new or changed, chunks it and BM25-tokenizes every chunk. par_iter().map().collect() preserves walk order.
• Phase 2 runs serially in walk order and does everything that mutates the index: skipped-large accumulation, the display-path collision skip, take_previous_rows reuse, BM25 remove/add with the precomputed tokens (reindex_file now takes token lists), the manifest, and fresh-row bookkeeping. Embedding is still one batched pass. Unchanged files drop their bytes in phase 1. If reuse fails anyway (it shouldn't, since the manifest range is bounds-checked first), the file is re-read and chunked inline.
• Worker count comes from the new CreateIndexOptions::index_workers: None → CSP_INDEX_WORKERS (a positive integer; anything else is ignored) → rayon default. 1 runs serially on the calling thread with no pool. The setting is documented in both READMEs and on the docs CLI page.
• Grammar safety (chunking/core.rs):
  • Resolved grammars are cached per language, so the language pack's global DOWNLOAD_CACHE_LOCK is not taken on every file. Failures are not cached, so a transient download error is still retried as before.
  • set_language, parse and parser drop run under one lock per grammar, keyed by the grammar's own name so aliases share it. A grammar whose external scanner keeps global state is never run concurrently with itself, but different languages still parse in parallel.
• Output is identical for any worker count. The new tests check this.

Bench (cargo bench -p code-search-please --bench core -- index_build, 300 generated ts/py/rs/go files, Model::Stub, 8-core macOS)

case	time
index_build/300_files_serial (workers = 1, the pre-change path)	199 ms (172.9 to 230.2 ms)
index_build/300_files_parallel (rayon default, 8 threads)	75.9 ms (75.0 to 77.0 ms)

That is about 2.6× faster. Parsing is serialized per grammar, so on a repo dominated by one language the speedup comes from read, hash, merge and tokenize, not from parsing.

E2E

I used a debug csp with the real minishlab/potion-code-16M-v2 model and a scratch HOME for each run, and ran csp search "parallel indexing" . --top-k 3 three ways: cold, warm, and cold in a fresh HOME with CSP_INDEX_WORKERS=1. The three JSON outputs are identical. So are the persisted cache files (bm25.json, chunks.json, manifest.json, vectors.bin, args.json) for the parallel and serial cold builds.

Follow-ups (not in this PR)

• Same-language parses still run one at a time. A per-thread parser pool, or an allowlist of grammars whose scanners are known to be safe, could lift that once each scanner is checked.
• The pool is built for each create_index_from_path call. A shared lazy pool could help long-lived MCP/SDK processes.

Related issue

Closes #115

Checklist

• PR title follows Conventional Commits
• Tests added or updated, and the suite passes (cargo test --workspace: 343 lib passed, 4 ignored; 27 CLI passed)
• Lint/format pass (cargo fmt, cargo clippy --all-targets --all-features -D warnings, bun run lint)
• Documentation updated if behavior changed (CSP_INDEX_WORKERS in README.md, README.ko.md, docs/cli.mdx)
• No breaking change, or a BREAKING CHANGE: note is included (adds a public field to CreateIndexOptions; callers that build it with ..CreateIndexOptions::new(model) are unaffected)

---

Summary by cubic

Parallelizes per-file indexing work in create_index_from_path, cutting a cold build of 300 files from ~199 ms to ~76 ms (≈2.6×) on an 8-core machine.

• Read, hash, chunk, and BM25 tokenization now run on a local rayon pool in batches of 256 files to bound memory; BM25 mutation and manifest assembly stay serial in walk order, so the index is identical for any worker count.
• Worker count is set by the new CreateIndexOptions::index_workers field or CSP_INDEX_WORKERS (1 = serial on the calling thread, default = one per CPU).
• Resolved grammars are cached per language and parsing is serialized per grammar because some external scanners keep process-global state; failures are not cached, so transient download errors still retry.
• Added a public field to CreateIndexOptions; callers using ..CreateIndexOptions::new(model) are unaffected.

Closes #115. Tests assert serial and parallel builds (cold, incremental, and concurrent first-time grammar resolution) produce identical chunks, manifest, vectors, BM25 documents, and query scores; the benchmark pins the file-size limit so the comparison is stable.

^{Written for commit 6bc0b3d. Summary will update on new commits.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 41 complexity · 2 duplication

Metric	Results
Complexity	41
Duplication	2

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request introduces parallel per-file indexing (reading, hashing, chunking, and tokenizing) using the Rayon library to improve performance, which can be configured via the new CSP_INDEX_WORKERS environment variable. It also implements caching and per-grammar locking for tree-sitter parsers to avoid lock contention and ensure thread safety with external scanners. The review feedback identifies a medium-severity issue in crates/csp/src/chunking/core.rs, where bypassing Mutex poisoning via into_inner() could allow threads to access corrupted tree-sitter parser states after a panic, potentially leading to undefined behavior. It is recommended to propagate the panic using unwrap() instead.

[codecov]

Codecov Report

❌ Patch coverage is 83.14607% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/csp/src/chunking/core.rs	58.92%	23 Missing ⚠️
crates/csp/src/indexing/create.rs	94.21%	7 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• 300_files_parallel

✅ 5 untouched benchmarks
🆕 2 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕 ⚠️	300_files_parallel	N/A	448.8 ms	N/A
🆕	300_files_serial	N/A	441.6 ms	N/A

---

_{Comparing amondnet/115-parallel-indexing (6bc0b3d) with main (dfa7193)}

[cubic-dev-ai]

cubic analysis

All reported issues were addressed across 11 files

Linked issue analysis

Linked issue: pleaseai/code-search#115: parity(#295): parallelize per-file read, hash, chunk, and tokenize during indexing

Status	Acceptance criteria	Notes
✅	Parallelize per-file reads, hashes, chunking, and tokenization	prepare_file does this in the Rayon phase; BM25 updates and manifest assembly remain serial in walk order.
✅	Bound parallel work in batches	The indexing loop prepares files in batches of 256 before applying results.
✅	Keep index output identical across worker counts	Tests compare chunks, vectors, manifest, BM25 documents, saved BM25 data, and query scores across serial and parallel builds.
✅	Benchmark cold indexing and record before-and-after timings	The new 300-file benchmark compares serial and parallel builds; the PR reports timings of 199 ms and 75.9 ms.

Architecture diagram

sequenceDiagram
    participant CLI as CLU/Entry Point
    participant Index as create_index_from_path
    participant Rayon as Rayon Thread Pool
    participant FS as File System
    participant Cache as Cache/Previous Index
    participant Chunker as Chunking Service
    participant Grammar as Grammar Registry
    participant Tokenizer as BM25 Tokenizer
    participant Writer as Index Writer

    Note over CLI,Writer: Parallel Indexing Architecture (Phases 1 & 2)

    CLI->>Index: create_index_from_path(path, options)
    Note over Index: Read CSP_INDEX_WORKERS or use index_workers option
    Index->>Index: index_pool(workers)

    alt workers == 1
        Note over Index: Serial mode (no pool)
    else workers > 1 or default
        Index->>Rayon: Build local ThreadPool
    end

    Note over Index: Phase 1 (Parallel, batches of 256 files)
    loop For each batch of files
        Index->>Rayon: par_iter().map(prepare_file)
        Rayon->>FS: metadata(file)
        FS-->>Rayon: file size
        alt size > MAX_FILE_BYTES
            Rayon-->>Index: TooLarge(display_path)
        else size <= MAX_FILE_BYTES
            Rayon->>FS: read(file)
            FS-->>Rayon: bytes
            Rayon->>Rayon: sha256_hex(bytes)
            Rayon->>Cache: Check previous manifest and bounds
            alt unchanged
                Note over Rayon: Drop bytes, mark Unchanged
                Rayon-->>Index: Ready(Unchanged)
            else new/changed
                Rayon->>Chunker: chunk_and_tokenize(bytes)
                Chunker->>Grammar: grammar_for(language)
                alt grammar not cached
                    Grammar->>Grammar: resolve_language(language)
                    Note over Grammar: Download/lookup via language pack
                    Grammar->>Grammar: Cache resolved grammar
                end
                Chunker->>Chunker: Parse with grammar lock
                Note over Chunker: Lock per grammar (aliases share)
                Chunker-->>Rayon: chunks
                Rayon->>Tokenizer: tokenize chunks
                Tokenizer-->>Rayon: tokens
                Rayon-->>Index: Ready(FreshChunks)
            end
        end
    end

    Note over Index: Phase 2 (Serial, walk order)
    loop For each Prepared item
        Index->>Index: Process in walk order
        alt TooLarge
            Index->>Index: Accumulate for warning
        else Unchanged
            Index->>Cache: take_previous_rows
            alt reuse fails
                Index->>FS: re-read file
                Index->>Chunker: inline chunk
            end
        else Fresh
            Index->>Index: Detect collisions
            Index->>Index: reindex_file(tokens)
            Index->>Writer: Write chunks, manifest entries
        end
    end

    Index->>Index: Embedding pass (batched)
    Index-->>CLI: CreateIndexResult

Loading

<sub>Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with the earlier non-blocking concern about token memory still open.

Summary

This PR runs per-file indexing work in parallel while applying index changes in file order.

Since the last review:

• Index builds prepare files in parallel without changing the resulting index.
• Grammar lookups are reused, while parser runs stay safe across files.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Files in walk order] --> B[Batches of up to 256 files]
  B --> C[Parallel read, hash, chunk, and tokenize]
  C --> D[Apply results serially in walk order]
  D --> E[Reuse unchanged rows and update BM25]
  E --> F[Embed fresh chunks]
  F --> G[Completed index]

Loading

_{Reviews (2) · Last reviewed commit: "test(index): pin the bench size limit an..." · Reviewed by Greptile}

[amondnet]

/gemini review

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
81.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[gemini-code-assist]

Code Review

This pull request introduces parallel file indexing (reading, hashing, chunking, and tokenizing) using the Rayon library, with worker counts configurable via the CSP_INDEX_WORKERS environment variable. It also adds caching for tree-sitter grammars to prevent lock contention during parallel processing. A compilation issue was identified in crates/csp/src/indexing/create.rs where a temporary Vec returned by walk_files is dropped while its borrowed iterator is still in use; binding this to a local variable resolves the error.