parity(#295): parallelize per-file read, hash, chunk, and tokenize during indexing #115

Summary

Upstream semble #295 (0e5e028) parallelizes the per-file work of indexing. A process pool (4 workers, used only when at least 200 files need chunking, with at most 256 files in flight) reads, chunks, and BM25-tokenizes each file. The main process applies the results in order, so the manifest and chunk order stay deterministic. SEMBLE_INDEX_WORKERS controls the worker count.

csp status

create_index_from_path (crates/csp/src/indexing/create.rs:254-324) is fully serial per file: metadata → fs::read → sha256_hex → chunk_source (tree-sitter parse) → tokenize(enrich_for_bm25(..)) → BM25 insert. crates/csp has no rayon, par_iter, or thread fan-out; rayon is only a transitive dependency. Embedding is already batched, so this is the remaining serial stage of a cold index build.

Proposal

• Phase 1, parallel (par_iter().map().collect() preserves order): read, hash, and, for changed files, chunk and tokenize.
• Phase 2, serial: BM25 mutation and manifest assembly.
• Process files in batches to bound memory, like upstream's 256-in-flight limit.
• Check that tree_sitter_language_pack::get_language (chunking/core.rs) is cheap and not contended across threads.
• Use RAYON_NUM_THREADS (or a CSP_INDEX_WORKERS knob) as the worker-count control.

Acceptance

• An index-build benchmark exists (crates/csp/benches covers only tokenize/rerank today). Record a before/after cold-index time on a mid-size repo.
• Output is identical: the same chunks, order, manifest, and bm25.json as the serial path.