Repository navigation
libpython: Speed up grass.lib import by registering $GISBASE/lib - #7605
Conversation
…/grass/grass_main/dist.x86_64-pc-linux-gnu/lib
nilason
left a comment
nilason
left a comment
There was a problem hiding this comment.
I can't see how this would break something, so let's try it out.
wenzeslaus
left a comment
wenzeslaus
left a comment
There was a problem hiding this comment.
The performance gain is crucial and the code placement is right. I did some AI/search to understand more here, specifically why this is even needed when we set LD_LIBRARY_PATH anyway (or even when we would do rpath): The mechanism this is influencing is the ctypesgen loading mechanism, so this works well given how ctypesgen is implemented to deal with cross-platform issues. The standard Linux (or any OS) library loading comes later in the process.
In case anyone is interested, here is an AI summary of my conversation with Google search AI/Gemini:
When LD_LIBRARY_PATH is already correctly set before Python starts, the Linux kernel loader has no trouble resolving deep dependencies. The core issue is that ctypes.util.find_library() prioritizes running /sbin/ldconfig -p, a system tool that completely ignores LD_LIBRARY_PATH and only scans standard system directories. When ldconfig fails to find your GRASS libraries in the system cache, Python enters a slow fallback routine that spawns heavy external shell subprocesses (gcc and objdump) to locate the file, creating a massive performance bottleneck for every single library imported.
Your add_library_search_dirs addition acts as a direct shortcut that short-circuits this broken Python loop using basic, instantaneous file-presence checks. By finding the entry-point library instantly, it skips the expensive find_library() routine entirely and hands ctypes.CDLL() an absolute path (like /path/to/grass/lib/libgrass_gis.so). Because your LD_LIBRARY_PATH is already active in the background, the Linux kernel loader seamlessly takes over from there, resolving all the deeper nested dependencies perfectly and instantly.
…eo#7605) Register $GISBASE/lib (where the GRASS shared libraries are installed) with the ctypes loader before any wrapper module loads its library. The loader searches these directories first, so it finds the libraries by absolute path and never reaches its ctypes.util.find_library() fallback. That fallback is expensive on Linux, where find_library() spawns the toolchain (ldconfig, gcc, ld, objdump) once per library; for the chain pulled in by grass.temporal that was about 90 subprocesses and the bulk of the import time. The libraries are still resolvable through the runtime linker path, so this is a startup optimization, not a correctness fix; outside a GRASS session GISBASE may be unset, in which case the loader keeps its original behavior.
Context is #7585 (import tgis takes too long). We found out that for Mac it is much faster, which triggered this solution. Claude debugged it:
Root cause
GRASS libraries live in
$GISBASE/lib, which the ctypes loader does not search before falling back toctypes.util.find_library(). On Linux that fallback shells out toldconfig/gcc/ld/objdumponce per library (~90 subprocesses for thegrass.temporalchain). macOS is unaffected because itsfind_library()spawns nothing.Fix
Register
$GISBASE/libwith the loader ingrass/lib/__init__.pyso libraries resolve by absolute path and the fallback is never reached. Gated onGISBASE, so out-of-session imports are unchanged.Results
Cold
import grass.temporalon Linux: ~2560 ms / 90 subprocesses → ~200 ms / 0. ~13x faster, matching macOS.@nilason I would appreciate your review on this. I tested the fix, seems fine, but I don't know if there are any unintended consequences.