Skip to content

tests: Cover the current grass.lib loading behavior - #8093

Merged
wenzeslaus merged 13 commits into
OSGeo:mainfrom
wenzeslaus:lib-loading-tests
Oct 27, 2026
Merged

wenzeslaus merged 13 commits into
OSGeo:mainfrom
wenzeslaus:lib-loading-tests

Conversation

@wenzeslaus

Copy link
Copy Markdown
Member

This adds tests which pin the current behavior of making the GRASS C libraries (grass.lib) available, without changing any behavior. grass.lib works with a session set up by init even in an environment without the dynamic library search path variable. A session does not load the C libraries into the process, so sessions using only tools never pay for the GRASS C stack. Setting the platform's search path variable before the process starts makes grass.lib work without a session in any layout, including an installation moved away from the prefix it was configured with.

The init function docstring is corrected accordingly. It claimed since 2015 that ctypes-based libraries are not usable after init, and now says that both importing and using grass.lib depend on init and when the import works. The whole situation is complex with things working differently on different platforms even across Linux distros.

The new CI steps run these tests as standalone pytests on all platforms and, unlike the other pytest steps, do not export the search path variable or add the library directory to PATH, so the tests see only the environment they construct themselves. I have two other PRs in prep which will modify the behavior and these tests.

Details (AI generated)

Two mechanisms behind the import

Importing a grass.lib module in a running process needs two things from two different mechanisms.

  1. The ctypes loader must find the library the module loads. It looks in directories it learns from the environment at the moment of the lookup, so the variables init writes into os.environ are enough. init sets the search path variable for the tools it starts as subprocesses, and the loader in the same process benefits from it as a side effect. The GISBASE registration from libpython: Speed up grass.lib import by registering $GISBASE/lib #7605 finds the same file faster and does not change this (the init-then-import case already worked with the GRASS 8.3 distribution package, which predates libpython: Speed up grass.lib import by registering $GISBASE/lib #7605).
  2. The dynamic linker must resolve the GRASS libraries that library depends on. It reads the search path variable only at process startup, so the variable set by init comes too late, and the dependencies resolve only through the search path recorded in the libraries at build time, which is valid for an installation at the prefix it was configured for.

For an installation at its prefix, init provides the first and the build-time search path the second, so init looks sufficient. For a moved installation, init still provides the first, but nothing in the running process can provide the second. Setting the variable before the process starts provides both, which is why that setup, used by CI and the test instructions, works in every layout and hid the difference.

What is pinned and why it holds

Behavior Mechanism
Import and use of grass.lib after init (no search path variable in the environment) init writes GISBASE and the search path variable into os.environ of the running process for its subprocesses; the ctypes loader in the same process reads them at lookup time and finds the library a grass.lib module loads, while the library dependencies resolve through the search path recorded in the libraries at build time (RUNPATH on Linux, install names on macOS, the loaded DLL's own directory on Windows). Verified down to GRASS 8.3 distribution packages.
A session does not load the C libraries Nothing in setup loads libraries; checked through /proc/self/maps in a subprocess (Linux).
The search path variable set at process start works everywhere, even for a moved installation The dynamic linker reads the variable at process startup for the library dependencies, and the ctypes loader scans the same variable for the top-level library. This is the manual setup used by the test instructions, the CI steps, and the Docker images.

On macOS, the variable GRASS knows (LD_RUN_PATH) is read by the ctypes loader but ignored by dyld, so the manual-setup test additionally sets DYLD_LIBRARY_PATH, documenting in executable form what the real manual escape hatch per platform is.

Why the new CI steps do not export the variable

Every existing pytest step exports LD_LIBRARY_PATH (or adds the library directory to PATH on Windows) before running, and the documented test setup instructs developers to do the same. That makes grass.lib always work in front of developers and in CI, which also means the layouts in which it fails never show up there. The new steps keep the environment clean and let the tests state their own requirements, so later changes to the loading have an honest baseline to diff against. The relocated-installation steps (Linux, macOS) run only the manual-setup test for now, because nothing else can pass there today. The clean-environment fixture removes the dynamic library search path variables and, on Windows, the PATH directories containing a GRASS core library, so running the tests from a GRASS session shell does not make them pass for the wrong reason.

Test placement

Session behavior is tested in python/grass/script/tests (init lives in grass.script.setup), plain loading without a session in python/grass/app/tests (grass.app.runtime is the home of the runtime setup machinery). Test module basenames follow the unit-name convention.

The subprocess fixtures are shared through a new python/conftest.py rather than duplicated in both test directories. The obvious place, python/grass/conftest.py, is not usable: python/grass is a package, so pytest imports a conftest.py there as grass.conftest, which imports the grass package from the source tree and shadows the installed one for the whole test process (verified: grass.__file__ then points to the source tree and grass.app.resource_paths holds the unconfigured template values). The python directory is not a package, so the conftest there is imported on its own and the installed grass package stays in use. pytest collects conftest files from the directory of each test file up to the root directory, so the standalone runs with only the two test files as arguments still get the fixtures, from the repository root as well as from any other working directory with absolute paths. Moving the tests themselves to a common directory is not needed for that.

Suggested squash-merge commit message

tests: Cover the current grass.lib loading behavior
Add tests which pin the current behavior of grass.lib without changing
any behavior. grass.lib can be imported and used with a session set up
by init in an environment without the dynamic library search path
variable (on Windows, the C libraries do not see the session), a session
does not load the C libraries into the process, and the search path
variable set before the process starts makes grass.lib work without a
session even for an installation moved away from its prefix.

The import after init works because the ctypes loader reads the search
path variable, which init sets for its subprocesses, from the
environment of the running process to find a library, while the dynamic
linker resolves the library dependencies through the search path
recorded at build time. For a moved installation, the variable is needed
before the process starts because the dynamic linker reads it only at
startup.

The tests construct the environment of their subprocesses themselves.
New CI steps run them standalone on Linux, macOS, and Windows without
the search path setup done by the other pytest steps, and against a
relocated installation on Linux and macOS. The shared subprocess
fixtures are in python/conftest.py because a conftest.py in python/grass
would make pytest import the grass package from the source tree.

Correct the init docstring, which claimed since 2015 that the libraries
used through ctypes are not usable after init. It now says that both
importing and using grass.lib depend on init, that grass.lib sees only
a session in the global environment, and when the import works.

The changes and this description were prepared with help of Claude Code (Claude Fable 5 and Claude Opus 5.5) and reviewed by me.

Pin the current behavior of grass.lib in a session set up by init in an
environment without the dynamic library search path variable: the
modules can be imported after init and the libraries see the session
(except on Windows, where the C libraries, built with a different C
runtime than Python, do not see variables set in the running process).
Also pin that a session does not load the C libraries into the process,
so sessions using only tools never pay for the GRASS C stack and its
dependencies. The tests construct the environment of their subprocess
themselves, so they are independent of how the test process was started.

The subprocess fixtures are in a new python/conftest.py, shared with the
grass.app tests, and not in python/grass/conftest.py: python/grass is a
package, so pytest would import a conftest.py there as grass.conftest
and with it the grass package from the source tree instead of the
installed one. Standalone runs with only the test files as arguments
still pick up the shared fixtures.
Pin the manual setup used by scripts, test instructions, and container
images: with the platform's dynamic library search path variable set
before the process starts, grass.lib works without any session and in
any layout, including an installation moved away from the prefix it was
configured with. The test takes the variable name from the runtime
(RuntimePaths) and sets DYLD_LIBRARY_PATH additionally on macOS, where
the variable known to GRASS (LD_RUN_PATH) is read by the ctypes loader
but ignored by the dynamic linker.
Run the new tests in steps which do not export the library search path
variable, unlike the other pytest steps, so that the steps pin what
works without that setup. On Linux and macOS, run the manual-setup test
also against an installation moved away from the prefix it was
configured with, where the search path recorded in the libraries at
build time cannot help; only the test which provides the variable
itself can pass there today. On Windows, the library directory is
deliberately not added to PATH in the new step.
The docstring claimed since 2015 that the libraries used through ctypes
are not usable after init. They are, for an installation which is at
the prefix it was configured for: the libraries find each other through
the search path recorded at build time, and the ctypes loader reads the
variables set by init from the environment of the running process. The
remaining limitation is an installation moved away from its prefix,
where the variable must be set before the process starts.
Make the clean subprocess environment free of all GRASS library search
paths: also remove LD_RUN_PATH (the variable GRASS uses on macOS) and,
on Windows, the PATH directories containing a GRASS core library, so
that running the tests from a GRASS session shell does not make them
pass for the wrong reason.

Make the Windows xfail of the usable-after-init test strict, so the test
reports when the C libraries start seeing the session on Windows instead
of passing silently.
The fixture removes the dynamic library search paths in general, not
paths specific to GRASS (only on Windows it removes just the PATH
directories with the GRASS libraries, as the comment there says).
The C libraries read the session from the environment of the process,
so a session set up with a custom env is not visible to grass.lib. This
is the current behavior, so document it together with the rest of the
corrected init docstring.
…tring

State first that both importing grass.lib and using it depend on init,
and then when the import works, without describing how grass.lib finds
its libraries.
@github-actions github-actions Bot added windows Microsoft Windows specific macOS macOS specific CI Continuous integration Python Related code is in Python libraries tests Related to Test Suite labels Oct 21, 2026
Add the copyright lines required by the style guide to the files added
in this series: the author and the GRASS Development Team before the
license identifier.
The manual-setup test prepended the library directory to the variable
GRASS knows but replaced DYLD_LIBRARY_PATH on macOS. Prepend to both,
so the two are handled the same way and an existing value is kept.
… on Windows

On Windows, the C libraries do not see environment variables set in the
running process (the strict xfail of the usable-after-init test), so a
session set up by init is invisible to grass.lib calls there. Say so in
a separate paragraph of the init docstring, which stays the same while
the import behavior described above it changes.
Say that grass.lib functions see a session set up by init only in
processes started afterwards, such as subprocesses, instead of the
ending about a session set up before the process starts.
A process started after init has the session in its environment from
its start, so the C libraries see it on all platforms, including
Windows, where they do not see a session set up in their own process.
This is the subprocess that the Windows skip in lib/gis/tests/
lib_gis_env_test.py refers to.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Continuous integration libraries macOS macOS specific Python Related code is in Python tests Related to Test Suite windows Microsoft Windows specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants