feat(dataflow): vendored typed_ast-free scalpel as the default L4 oracle#111
Open
rahlk wants to merge 6 commits into
Open
feat(dataflow): vendored typed_ast-free scalpel as the default L4 oracle#111rahlk wants to merge 6 commits into
rahlk wants to merge 6 commits into
Conversation
Vendor the 9-module SSA/cfg/core slice of python-scalpel 1.0b0 (Apache-2.0, typed_ast-free) into codeanalyzer/dataflow/scalpel/, making ScalpelAliasOracle the shipping-default L4 oracle on Python 3.9-3.14+ with no external python-scalpel/typed_ast dependency. Type-based oracle demotes to the runtime safety net. Includes feasibility evidence, oracle rewiring, the L4 precision behavior change, and the testing plan.
Repoint ScalpelAliasOracle.from_function at the vendored, typed_ast-free
codeanalyzer.dataflow.scalpel slice instead of the external python-scalpel
package, and drop the now-dead ImportError fallback branch in
make_alias_oracle -- Scalpel can no longer be "absent" since it ships in
the package. TypeBasedAliasOracle remains the runtime safety net for a
per-callable Scalpel build failure only.
Reframe the two L4 tests whose premise this repoint invalidates:
test_make_alias_oracle_falls_back_when_scalpel_absent now forces a
per-callable build failure instead of poisoning sys.modules["scalpel"]
(which no longer affects the vendored import path), and
test_scalpel_oracle_copy_chain drops its pytest.importorskip("scalpel")
guard since the vendored oracle is always present.
…ved [scalpel] extra)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Vendor a minimal,
typed_ast-free slice ofpython-scalpel 1.0b0intocodeanalyzer/dataflow/scalpel/and makeScalpelAliasOraclethe shipping-default L4 may-alias oracle on every supported Python — with no externalpython-scalpel/typed_astdependency. The type-based oracle demotes to a pure runtime safety net.Why
python-scalpelhard-depends ontyped_ast, an abandoned package whose last release (1.5.5) has no wheel for Python 3.12+ and fails to build from source there. Sopip install python-scalpelfails on 3.12/3.13/3.14, and scalpel — the primary L4 points-to oracle — was an optional extra that silently degraded to the coarser type-based oracle on modern Python.typed_astis imported by exactly one scalpel module (typeinfer), which this analyzer never uses. Vendoring the small slice the oracle actually needs removes thetyped_astwall entirely.How
codeanalyzer/dataflow/scalpel/— exactly the 9-moduleSSA/cfg/coreclosure the oracle loads (verified free oftypeinfer/typed_ast), copied verbatim from1.0b0except one patch:cfg/model.py's module-loadimport graphviz as gvis guarded so the unused visualization code no longer forces agraphvizdependency.LICENSE+ a provenanceREADME.mdin the vendored dir + a top-levelNOTICEentry.scalpel_oracle.pyimports the vendored path; the dead "scalpel not installed" fallback branch is removed (scalpel can't be absent now); the per-callable build-failure and per-query unresolved-path fallbacks toTypeBasedAliasOracleremain, so themay_aliasinterface stays total and never raises.astoradded to core (a genuine runtime dep of scalpel's SSA; pure-Python, universal); the[scalpel]extra removed;requires-pythonunchanged at>=3.9.Behavior change
On Python 3.12+ — and for anyone who never installed the old
[scalpel]extra — L4'sprov:["points-to"]DDG edges were derived from the type-based over-approximation; they are now scalpel-precise (a tighter subset). Theprov:["ssa"]edges are unchanged, so the monotonicity invariantL3 ⊆ L4still holds. Users who already had the[scalpel]extra on ≤3.11 see no change (same code). Recorded inCHANGELOG.md.Testing
test_v2_l4,test_v2_l4_summary,test_dataflow_sdg,test_dataflow_defuse) and thetest_l3_subset_of_l4monotonicity gate pass, now genuinely exercising scalpel-derived points-to. Newtest/test_vendored_scalpel.pygates thetyped_ast-free property (imports + computes SSA with notyped_ast), asserts scalpel is the default oracle, checks determinism, and includes a vendored-vs-upstream SSA fidelity check (skips where upstream can't install).release.yml) runs on Python 3.10, where both PyCG and the vendored scalpel work; the removed[scalpel]extra doesn't affect its.[neo4j]install.Note on Python 3.14
The analyzer's own default dev interpreter here is 3.14, where PyCG is independently broken (
ast.Num.nwas removed in 3.14), so the CLI L4 pipeline can't run end-to-end on 3.14 until that unrelated PyCG issue is addressed. This PR'styped_ast-free property is proven on 3.14 in isolation; the full L4 verification runs on 3.12. The scalpel-vendoring change itself is Python-version-agnostic.