perf(extract): parallelize the Python resolution tail and memoize path work - #3214
perf(extract): parallelize the Python resolution tail and memoize path work#3214toharush wants to merge 1 commit into
Conversation
…h work (Graphify-Labs#3008) extract() is 4.4x faster cold and 5.7x faster warm on a 501-file Python-heavy corpus, and 4.0x faster at 8,000 files on a large C#/TypeScript monorepo, with byte-identical output. detect() is 1.35x faster on a 38,666-file scan emitting an identical path set. The extraction pool now runs on a forkserver context with graphify.extract preloaded, so per-worker interpreter startup is paid once per run instead of once per worker; Windows, which has no forkserver, keeps the default context and the existing BrokenProcessPool fallback. The two Python resolution hotspots -- fact collection and the identifier walk in _resolve_cross_file_imports -- were 1.47s of a 2.01s serial tail and are now per-file functions returning plain str/int payloads that a pool produces and the parent stitches back together in input order. That order is load-bearing, since the fact lists and the per-statement import order flow into emitted edge order, and the parent keeps what only it can hold: the module-stem index, the local-symbol name maps, and the edge emission. The two changes only pay off together -- under spawn the extra pools re-import the module per worker and cost more than the walks they parallelize -- so the split is gated on a forkserver being available and on at least 60 Python files, and a pool that fails for any reason discards the whole pass and the parent redoes it in-process, where the payloads are identical. Path resolution and one Python parse per file are now memoized for the life of one run, cleared per run by both extract() and detect() so a long-lived watch process cannot carry one cycle's symlink targets or source into the next. The import_from_statement and call scans are compiled tree-sitter queries with captures re-sorted to pre-order and a full-walk fallback when a cursor hits its match limit. C# type-reference arbitration indexes sourceless nodes by label once instead of scanning the graph per unresolved reference. Ignore matching compiles its per-pattern derivations once per scan instead of parsing them once per path, with literal and anchored patterns skipping the general fnmatch cascade. At 16,000 files the AST phase becomes memory-bound -- 6.9s at 8,000 files to ~140s at 16,000, in both the before and after trees -- so the extraction-side win is gone there and only the ~6x resolution-tail win survives. Sizes above 16,000 were not measured.
toharush
changed the title
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Speeds up cache save and directory scans by replacing per-item and per-pattern recomputation with memoized work. _relativize_source_files_in now resolves each distinct source_file string once per payload instead of once per node, and save_cached deep-copies via _json_deepcopy (dict/list/scalar only) rather than copy.deepcopy, since the payload is JSON-shaped and about to be serialized. Cache path resolution across cache.py switches from Path(...).resolve() to the shared resolve_cached, and detect.py drops the hand-rolled pattern-parse cache in favor of lru_cache-backed _match_anchored_ignore_pattern that matches anchored ignore patterns by slicing normcased strings instead of building Path objects per entry.
Worth a look
- _py_fact_payloads indexed by Python-only slots but merge receives full paths list —
graphify/extract.py:6387· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- _json_deepcopy leaves tuple-contained dicts shared —
graphify/cache.py:842· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Compiled ignore cache reuses stale rules after in-place pattern replacement —
graphify/detect.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Compiled-pattern cache keyed on id() may reuse stale entry after list mutation —
graphify/detect.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Compiled ignore cache ignores in-place pattern mutations —
graphify/detect.py:1536· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 3856 functions depend on the 659 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 517 callers, 48 callees - new:
_rebuild_code()— 108 callers, 50 callees - new:
build_from_json()— 192 callers, 18 callees - new:
detect()— 108 callers, 16 callees - new:
build_merge()— 53 callers, 13 callees - new:
save_semantic_cache()— 58 callers, 9 callees - new:
to_obsidian()— 36 callers, 13 callees - new:
_extract_generic()— 18 callers, 25 callees - …and 145 more — each is listed as a finding
Verification — 3856 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 3440 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify cache\_dir.
The verifier did not have enough to check cache\_dir, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify cached\_word\_count.
The verifier did not have enough to check cached\_word\_count, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify file\_hash.
The verifier did not have enough to check file\_hash, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify prune\_semantic\_cache.
The verifier did not have enough to check prune\_semantic\_cache, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_relativize\_source\_files\_in.
The verifier did not have enough to check \_relativize\_source\_files\_in, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify save\_cached.
The verifier did not have enough to check save\_cached, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_stat\_index\_file.
The verifier did not have enough to check \_stat\_index\_file, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify detect.
The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_is\_ignored.
The verifier did not have enough to check \_is\_ignored, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_lexical\_relative.
The verifier did not have enough to check \_lexical\_relative, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `target` is annotated `Path` — outside the synthesizable primitive/collection set
No difference found (not proven): No behavior difference found in \_match\_anchored\_ignore\_pattern (not a proof).
The verifier ran both versions of \_match\_anchored\_ignore\_pattern on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_resolves\_under\_root.
The verifier did not have enough to check \_resolves\_under\_root, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract.
The verifier did not have enough to check extract, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `cache_root` is annotated `Path | None` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract\_csproj.
The verifier did not have enough to check extract\_csproj, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_extract\_parallel.
The verifier did not have enough to check \_extract\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract\_slnx.
The verifier did not have enough to check extract\_slnx, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_lang\_family.
The verifier did not have enough to check \_lang\_family, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `source_file` is annotated `object` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_lang\_is\_case\_insensitive.
The verifier did not have enough to check \_lang\_is\_case\_insensitive, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `source_file` is annotated `object` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_merge\_csharp\_partial\_class\_nodes.
The verifier did not have enough to check \_merge\_csharp\_partial\_class\_nodes, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_repoint\_python\_package\_imports.
The verifier did not have enough to check \_repoint\_python\_package\_imports, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 51 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_rescue\_js\_dynamic\_imports.
The verifier did not have enough to check \_rescue\_js\_dynamic\_imports, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_resolve\_python\_member\_calls.
The verifier did not have enough to check \_resolve\_python\_member\_calls, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
Could not verify: Could not verify \_xaml\_csharp\_class\_nodes.
The verifier did not have enough to check \_xaml\_csharp\_class\_nodes, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_xaml\_project\_root.
The verifier did not have enough to check \_xaml\_project\_root, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_resolve\_csharp\_type\_references.
The verifier did not have enough to check \_resolve\_csharp\_type\_references, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: non-vacuity: domain too small (only 1 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
· 5 grounded finding(s) anchored inline below; 148 more finding(s) on lines outside this diff (see the check run).
| return True | ||
|
|
||
|
|
||
| def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict: |
There was a problem hiding this comment.
detect()
fans out to 16 callees (efferent coupling); 108 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @@ -4965,7 +5006,7 @@ def _xaml_project_root(path: Path) -> Path: | |||
| def _xaml_csharp_class_nodes(path: Path) -> dict[str, list[dict]]: | |||
There was a problem hiding this comment.
_xaml_csharp_class_nodes()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return max(max_workers, 1) | ||
|
|
||
|
|
||
| def _extract_parallel( |
There was a problem hiding this comment.
_extract_parallel()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
| trees: dict[Path, tuple[bytes, object]] = {} | ||
|
|
||
| def _collect_export_facts(node, source: bytes, path: Path) -> None: |
There was a problem hiding this comment.
_collect_export_facts()
fans out to 12 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return _read_text(function_node, source) | ||
| return None | ||
|
|
||
| def _collect_python_file_facts(path: Path, root: Path) -> "dict | None": |
There was a problem hiding this comment.
_collect_python_file_facts()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
What this is
Performance work on
extract()anddetect(). No behavior change: everyharness below digests the result and fails if it moves, and all of them report
identical output before and after.
Measured, before vs after
"Before" is
680e3ed8, this branch's base — verified byte-identical togit show HEAD:<file>for all seven files touched here. Every number comes fromone A/B batch with the trees alternated rep by rep, each rep in a fresh
subprocess with a fresh temp cache root, so nothing inherits a warm AST cache, a
warm resolve cache, or a warm interpreter.
extract()— 501-file Python-heavy tree (this repo), 5 reps, median wall:watch/updatepath)extract()— scale curve on a large C#/TypeScript monorepo, seeded sample persize:
detect()— full scan of a 38,666-file monorepo emitting 48,407 paths, 3reps, median: 33.35s to 24.74s (1.35x), 0.689ms to 0.511ms per emitted
path, identical path set.
Where the time went
Split by phase,
astbeing the parallel extraction phase andpostthe serialcross-file resolution phase after it:
The serial resolution tail is where nearly all of this lands, which is also why
the warm path gains the most: on a filled cache nothing is extracted, so the run
is the tail.
The 16,000-file point underdelivers, and it is not fixed here
Read the
astcolumn across sizes: 6.9s at 8,000 files, then ~140s at 16,000.Twice the files, twenty times the time, in both trees. That is not a parsing
curve — it is the machine running out of RAM while holding a 303,000-node graph
plus a worker pool. Where that phase waits on memory rather than CPU, saving
each worker its module import buys nothing, and the after-tree measures a shade
slower there in both reps rather than faster.
So the honest shape: the extraction-side win is real and about 1.6x up to ~8,000
files on this hardware and gone by 16,000; the resolution-tail win holds at ~6x
at every size measured and is the entire 1.4x at 16,000. Sizes above 16,000 were
not measured and are not extrapolated. Making the AST phase scale past that is
separate work.
The changes
Extraction pool on a
forkservercontext, withgraphify.extractpreloaded. Under
spawn— the default on macOS and Windows — every worker is afresh interpreter re-importing the module, so that startup was paid once per
worker instead of once per run: measured worker CPU grew from 3.95s at 6 workers
to 4.65s at 16, and the pool phase got slower past 8 workers because the added
startup outran the added parallelism. A forkserver imports once and forks each
worker from that warm image. Windows, which has no forkserver, keeps the default
context and the existing
BrokenProcessPoolfallback.The two Python resolution hotspots moved out of the serial parent, where
they were 1.47s of a 2.01s tail. Both are one parse plus one tree walk per file,
over files the workers had already parsed, and only the cross-file merge
genuinely needs the global node table.
_collect_python_symbol_resolution_factsand the identifier walk inside
_resolve_cross_file_importsare now per-filefunctions returning plain
str/intpayloads that a pool produces and theparent stitches back together in input order. That order is load-bearing — the
fact lists and the per-statement import order flow into emitted edge order — and
the parent keeps what only it can hold: the module-stem index, the local-symbol
name maps (node ids exist only there, and only after the id remap), and the edge
emission.
These two only pay off together. Isolated in the same batch, the forkserver
alone is 3.59s to 2.79s and the pass split alone is 3.59s to 4.23s — under
spawn the two extra pools re-import the module per worker and cost more than the
walks they parallelize. So the split is gated on a forkserver actually being
available and on a corpus of at least 60 Python files, and a spawn-only platform
keeps both passes in the parent. A pool that fails for any reason discards the
whole pass and the parent redoes it in-process, where the payloads are
identical, so a broken pool costs time and nothing else.
Path resolution memoized for the life of one run.
Path.resolve()followssymlinks component by component, one
lstatper segment, and cross-fileresolution asked the same question about the same few thousand paths millions of
times: a profile showed 3.9M
realpathcalls issuing 59Mlstatsyscalls, with_source_keyalone 36% of the run. The caches key on paths with no mtimecomponent, so
extract()clears them per run exactly as it already did for thetsconfig caches, and
detect()now clears them at the start of a scan for thesame reason — otherwise a long-lived
watchprocess would carry the previouscycle's symlink targets into this scan.
One parse per Python file per run instead of three. The same file was parsed
in the worker's
_extract_python_rationaleand then again in each of the twoparent-side phases. The parent-side phases now share one parser and one tree per
file; that cache treats a file's contents as fixed for one run, so
extract()clears it on entry under the same contract as
clear_resolve_cache()and clearsit again as soon as the last consumer finishes, so the trees are not held past
the peak.
Node-type scans pushed into tree-sitter's C layer. The
import_from_statementand
callscans are compiled queries rather than full Python-side walks —measured 3.6x and 3.1x faster, where one walk visits 1.44M nodes to reach a few
thousand of interest. Captures arrive in match-completion order, so they are
re-sorted by start byte ascending then end byte descending, which reproduces
pre-order exactly (a parent always starts no later than its child and ends no
earlier); that order is observable, since it flows into emitted edge order. A
cursor that hits its match limit falls back to the full walk rather than acting
on a silently truncated capture set.
C# type-reference arbitration indexed instead of scanned. It minted a
dangling stub by scanning every node in the graph for a matching sourceless one,
once per unresolved reference — 194M dictionary lookups from 102,650 calls on a
32,000-file corpus. Sourceless nodes are now indexed by label once, and the
index registers the stubs it mints.
Ignore matching compiled per scan rather than parsed per path. Per-pattern
derivations — negation, directory-only, anchored, literal, pre-split segments —
are compiled once per scan, with the list extended in place as nested ignore
files are appended during the walk. Anchor membership becomes a normcased string
prefix test needing no
partstuple. Two shapes that dominate real ignore filesskip the general
fnmatchcascade: a literal name with no wildcard collapses toone set membership test, and an anchored pattern with no
**matches strictlypositionally. The
**matcher is a segment automaton rather than a memoizedrecursion whose key carried the path, so consecutive paths never shared an entry.
Smaller repeated work removed: the workspace-root walk (which reads and
JSON-parses every ancestor
package.json) is memoized per starting directory;JS/TS declaration, alias, and export collection share one depth-first pass
instead of three, 18M fewer node visits;
Path(...).suffixin two language-familypredicates becomes a memoized
splitext, 1.7M calls; module-specifier resolutionis memoized per directory, halving its 76,317 filesystem probes;
normalize_idis memoized (448,282 calls);
_module_stem_keyis memoized on the node id; theproximity tie-breaker's
PurePosixPathparse is memoized on the normalizedstring, still through pathlib so
.parent's handling of//,.components andbare filenames is unchanged; the cache-write path answers
os.path.relpathonceper distinct value instead of once per item (61,964 calls across 438 payloads),
and
copy.deepcopyof a payload bound forjson.dumpsbecomes a three-branchrecursive copy over dicts, lists, and immutable scalars.
Two candidates measured and rejected rather than shipped
CPython (0.299s vs 0.555s over the same corpus). The earlier generator win came
from
yield fromframe resumption, which plain recursive functions do not pay._resolve_cross_file_importsas a query plusa byte-order scope sweep is 1.3x slower (0.44s vs 0.56s) — the C-side query win
is more than consumed by the Python-side scope bookkeeping it needs.
A note on CPU time
Forkserver workers are forked from the server process, which makes them
grandchildren that the parent's
RUSAGE_CHILDRENcannot see. Measured CPU forthe 501-file corpus collapses from 11.44s to 1.01s across this branch, and
essentially all of that is the measurement losing sight of the workers, not work
disappearing. Wall time is the only figure that means the same thing on both
sides of this change, so wall time is the only figure reported.
Deliberately not in this PR
Tests, benchmark harnesses, changelog entries, and the measurement write-up are
held back at the author's request and will land separately. Worth flagging what
that leaves uncovered until they do:
checks them against a reference implementation of the previous semantics, and
without it a fast path that diverges from the general cascade fails silently
equivalence; both properties are invisible from graph output, and a
Pathordataclass creeping back into a payload breaks the pool at runtime, not at
import