Skip to content

g.extension: support alternative addon servers for MS-Windows users - #7383

Open
ecodiv wants to merge 12 commits into
OSGeo:mainfrom
ecodiv:g_extension_windows
Open

ecodiv wants to merge 12 commits into
OSGeo:mainfrom
ecodiv:g_extension_windows

Conversation

@ecodiv

@ecodiv ecodiv commented May 10, 2026

Copy link
Copy Markdown
Contributor

Motivation

On MS-Windows, g.extension downloads precompiled addon ZIPs from a hardcoded URL (http://wingrass.fsv.cvut.cz). There is no way to point Windows users at a different server. The url= option is documented as not supported on Windows, and the WinGRASS host is baked into install_extension_win(). The listing URL is similarly hardcoded to https://grass.osgeo.org/addons/<branch>/. Consequence:

  • Running an mirrow, good if the official one is not available or updated
  • Hosting addons that aren't in the official repository (e.g. organisation-specific tools, forks, addons not yet upstreamed).

This PR adds two settings that redirect both listing and install to a custom server, plus a per-call url= for one-off ZIP installs. No new parser options are introduced, and the default behavior is unchanged. For Linux users, there should be no change at all.

Changes

New: get_addons_base_url() helper, which resolves the base URL of the addon server with the following precedence:

  • ADDONS_BASE_URL set via g.gisenv set="ADDONS_BASE_URL=..." (persists across sessions, settable from the GUI's variables panel).
  • GRASS_ADDONS_BASE_URL shell environment variable (useful for CI / one-off shell sessions).
    Built-in default http://wingrass.fsv.cvut.cz.

Guards against gs.gisenv() failing outside an active GRASS session.

install_extension_win() honors the resolver and url=

  • The previously hardcoded "http://wingrass.fsv.cvut.cz/" base is replaced by get_addons_base_url().
  • The function now accepts a url parameter for Window users. If url= is given, it must be a full ZIP URL ending in .zip. The zip file must reside on a server with pre-compiled add-ons with the same structure as the default server.
  • Non-ZIP values for url= (e.g. GitHub repo URLs, base server URLs) are rejected with a error pointing the user at ADDONS_BASE_URL / GRASS_ADDONS_BASE_URL. This avoids platform-dependent reinterpretation of the same url= value.

What does not change

  • No new parser options. The g.extension GUI/CLI surface is unchanged on every platform.
  • Non-Windows code paths are untouched. Linux/macOS users see no behaviour change.
  • The default still is for Windows installs is to point at wingrass.fsv.cvut.cz and grass.osgeo.org/addons/grass8/ respectively.

Server-side

A custom server is expected to mirror the official WinGRASS layout:

{base}/grass{MM}/addons/grass-{M.M.P}/
    modules.xml
    <addon>.zip
    <addon>.md5sum
    ...

I have an example at https://ecodiv.earth/share. This contains Python scripts only. These were created by pulling the grass-addon repository and using a custom script to create the zipped addon files.

Testing and AI

I used AI to help me write and test the proposed changes. I tested the script on Linux (Ubuntu 24.04) and Windows 11. On Linux, no changes in behavior or options. On Windows, I installed addons setting ADDONS_BASE_URL=https://ecodiv.earth/share and by using the url=.. pointing directly at the zip file.

ecodiv added 2 commits May 10, 2026 11:22
Proposal to add the option to point at an alternative addon server for MS-Window users.
@github-actions github-actions Bot added Python Related code is in Python HTML Related code is in HTML module docs markdown Related to markdown, markdown files general labels May 10, 2026
@ecodiv

ecodiv commented May 10, 2026 •

Copy link
Copy Markdown
Contributor Author

You can test it using https://ecodiv.earth/share for 8.5.0 and 8.6.0dev

@ecodiv
ecodiv requested a review from wenzeslaus May 10, 2026 10:15
@ecodiv
ecodiv marked this pull request as draft May 10, 2026 10:16
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@ecodiv
ecodiv marked this pull request as ready for review May 13, 2026 11:45

@echoix echoix left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

Comment thread scripts/g.extension/g.extension.py
@ecodiv

ecodiv commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

For Linux it is already possible to provide an alternative URL, so this adds an option that is already available for Linux (for Mac, I don't know).

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

That is a good question. I don't think it is ideal, it is just the only way I could think of that would not require changes in the current interface, and which I thought would still be easy enough for users.

@echoix

echoix commented May 21, 2026

Copy link
Copy Markdown
Member

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

For Linux it is already possible to provide an alternative URL, so this adds an option that is already available for Linux (for Mac, I don't know).

Can windows be adapted to do it too if that’s enough?

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

That is a good question. I don't think it is ideal, it is just the only way I could think of that would not require changes in the current interface, and which I thought would still be easy enough for users.
No problem

@ecodiv

ecodiv commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

You're right that this isn't fundamentally Windows-only. The Linux flow also has a default location (grass.osgeo.org/addons/ for the listing, the OSGeo grass-addons GitHub repo for sources) that could be redirected the same way using the ADDONS_BASE_URL. The main reasons this PR focusses on Windows is that the binary-distribution layout makes the implementation trivial there. A Linux equivalent has to deal with multiple source mechanisms (git/zip/svn/local), which makes it (for me at least) more complicated.

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

I am not sure if it is the best way, but it was the best way I could think of that does not mess with the current interface and is still easy enough for users. It is still not as easily discoverable as I would like, so perhaps a dedicated parameter in g.extension itself would be better? It still should, in my opinion, expose the list of addons available on the selected server when using g.extension through the menu.

@ecodiv

ecodiv commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

For Linux it is already possible to provide an alternative URL, so this adds an option that is already available for Linux (for Mac, I don't know).

Can windows be adapted to do it too if that’s enough?

Yes, and this PR already does that, it also extends url= to Windows (for full .zip URLs, matching Linux's existing ZIP-URL handling).

Beyond that, the reason I think the base-URL mechanism matters specifically for Windows is operational: addon provision depends on precompiled binaries built by essentially one maintainer (as far as I understand), and there have historically been extended periods where addons weren't being updated. An alternative-server option gives Windows users a real fallback in those situations, which is less of a concern on Linux, where builds are from source.

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

That is a good question. I don't think it is ideal, it is just the only way I could think of that would not require changes in the current interface, and which I thought would still be easy enough for users.
No problem

@hellik

hellik commented May 21, 2026

Copy link
Copy Markdown
Member

If it’s something that essential to have, why would it be limited to Windows? Couldn’t it work for other platforms (to set a base url), even if they are t using pre-compiled addons?

For Linux it is already possible to provide an alternative URL, so this adds an option that is already available for Linux (for Mac, I don't know).

Can windows be adapted to do it too if that’s enough?

Yes, and this PR already does that, it also extends url= to Windows (for full .zip URLs, matching Linux's existing ZIP-URL handling).

Beyond that, the reason I think the base-URL mechanism matters specifically for Windows is operational: addon provision depends on precompiled binaries built by essentially one maintainer (as far as I understand), and there have historically been extended periods where addons weren't being updated. An alternative-server option gives Windows users a real fallback in those situations, which is less of a concern on Linux, where builds are from source.

Is tweaking around with env vars how we want this to work? It’s easier to implement, like done here.

That is a good question. I don't think it is ideal, it is just the only way I could think of that would not require changes in the current interface, and which I thought would still be easy enough for users.
No problem

On Windows, a pre-compilation is needed for C/C++ addons.

For python addons, "only" a related windows batch file (.bat) is needed.

Would it make sense to integrate such a bat-file creation function e.g. e.g. in g.extension? then, at least, python addons would be available from source also for windows; only C addons would need a pre-compilation on a server.

@ecodiv

ecodiv commented May 24, 2026

Copy link
Copy Markdown
Contributor Author

@echoix @hellik, what about limiting the scope to making the url parameter work for Windows users for now? This allows Window-users to install individual addons from elsewhere (on the condition that these addons are properly packaged). This brings this functionality on par with what is already available for Linux and seems like a small non-intrusive change.

The idea of hellik would complement this. I still would like the option to set a different base-server (I agree it would be great to have this for Linux as well) but I can move that perhaps to an addon?

@hellik

hellik commented May 24, 2026

Copy link
Copy Markdown
Member

@echoix @hellik, what about limiting the scope to making the url parameter work for Windows users for now? This allows Window-users to install individual addons from elsewhere (on the condition that these addons are properly packaged). This brings this functionality on par with what is already available for Linux and seems like a small non-intrusive change.

The idea of hellik would complement this. I still would like the option to set a different base-server (I agree it would be great to have this for Linux as well) but I can move that perhaps to an addon?

Sure, go ahead with your approach!

For the mid term, we should look at additional options.

@ecodiv

ecodiv commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

I propose to go ahead for now with this solution (I'll check the conflicts), so at least Window users have access to some of the addons (Python). Reason I am picking this up again and would like a solution, even if temporary and not ideal, is that teaching that depend on addons is basically impossible.

@echoix

echoix commented Oct 12, 2026

Copy link
Copy Markdown
Member

If having this merged allows you to unlock your use case, now, let's just merge it (when conflicts are solved)

@ecodiv
ecodiv requested a review from echoix October 14, 2026 05:41

@echoix echoix left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some contradictions in the docs, and the precedence of the variable vs settings the agent found (opus 5.5)

Details

Branch pr-7383 is checked out; reading the surrounding code in g.extension.py now to check how the new URL handling flows.

I've checked out PR #7383 locally as branch pr-7383. It changes 3 files under scripts/g.extension/ (.py, .md, .html). The idea is sound and the code change is small. But the PR has a file-mode regression, the documentation contradicts the code, the official server doesn't have the layout the docs promise, and the .md and .html files disagree.

I tested the server claims against the live servers. I didn't run the tool on Windows, and I haven't posted anything to GitHub.

Must fix

  1. The script loses its executable bit. g.extension.py goes from mode 100755 to 100644 (upstream main has 100755). This was probably caused by editing on Windows and needs to be reverted.

  2. The official WinGRASS server has no modules.xml or toolboxes.xml at the documented path. https://wingrass.fsv.cvut.cz/grass84/addons/grass-8.4.1/modules.xml returns 404, and so does toolboxes.xml. The directory has only ZIPs, .md5sum files and logs/.

    • The docs say an alternative server should use "the same directory layout as the official WinGRASS server … containing … a modules.xml listing". That isn't true, so an exact mirror of the official server would install fine but fail on g.extension -l.
    • The docs should state that modules.xml is an extra requirement.
    • With the override set, -l -t and -t installs also need toolboxes.xml, which the docs don't mention.
  3. The url= documentation contradicts the code.

    • The code (install_extension_win) now requires url to end in .zip, so the docs' own example url=http://example.com/.../r.example?format=zip is rejected on Windows.
    • The new text after that example says url= works on Windows "when pointing at a server with pre-compiled addons and … the same directory layout". That's wrong: the code downloads url directly as a single precompiled ZIP and never treats it as a server base.
    • The text should say that on Windows url must be a ZIP of a precompiled addon. A source ZIP from GitHub won't work.
  4. The .md and .html files disagree on the alternative server URL.

    • The .md uses https://ecodiv.earth/share/AddonServer, which works: …/AddonServer/grass85/addons/grass-8.5.0/modules.xml returns 200.
    • The .html uses https://ecodiv.earth/share in the server list and in the g.gisenv example. That's wrong: …/share/grass85/addons/ returns 404.
    • The cmd.exe example uses the wrong share URL in both files.

Should fix

  1. The AUTHORS entry renders on the same line as the one before it. The .md line before the new author is missing its two trailing spaces (the line break), while the .html has a <br>.
  2. Typos and inconsistencies in the docs:
    • "settings explained in this section are is honoured" ("are is" is a typo).
    • "redirect the addon server … once per session" contradicts "persists across GRASS sessions" two lines later.
    • There's a Unicode em dash ("—") where the style guide prefers ASCII.
    • The NOTES section now says both "only the official repository is working" and "it is now possible to point at an alternative server". The old sentence should be rewritten rather than appended to.
  3. The precedence order is questionable. The persistent gisenv setting overrides the shell variable GRASS_ADDONS_BASE_URL. Usually it's the reverse: the short-lived, more specific environment variable wins, which is what you'd want for the CI and one-off cases the docs describe. Worth asking the author why.
  4. except Exception in get_addons_base_url_override. g.extension always runs inside a session, so this broad catch (with a noqa) mostly hides real errors. Either catch only the specific error or drop the try.
  5. get_addons_base_url() adds little. It has one caller and is a one-line or, so it could be inlined.

Minor / for discussion

  • Behaviour change: on Windows, url= used to be silently ignored, so the official addon was installed. Now a GitHub or source URL fails with an error. That's better than silently ignoring it, but it should be mentioned in the PR description.
  • With -t and url= on Windows, every addon in the toolbox downloads the same ZIP. Toolbox installs on Windows were already broken before this PR (xmlurl is never passed), so this is low priority.
  • No tests: addons_server_url() and the precedence logic could get a small pytest in tests/, for example using monkeypatch on GRASS_ADDONS_BASE_URL with sys.platform patched.
  • The commit history includes "some tweaks" and two merges from main, so it should be squash-merged.

Do you want me to post this as a review comment on the PR, or prepare fixup commits for items 1–6?

- Clarified that an alternative server needs the ZIPs plus modules.xml (and toolboxes.xml for -t)
- Rewrote the url= section: on Windows url must point at a .zip of a single precompiled addon; use - ADDONS_BASE_URL to switch servers.
- Fixed the .html to use https://ecodiv.earth/share/AddonServer.
- few other fixes

In code:
- Flipped precedence, now GRASS_ADDONS_BASE_URL (env var) wins over the ADDONS_BASE_URL gisenv setting.
- Dropped the broad except Exception in get_addons_base_url_override() and inlined the one-line get_addons_base_url() wrapper.
- Behaviour change: on Windows url= was previously ignored; it now errors on a non-.zip URL.
- File-mode regression reverted
@ecodiv

ecodiv commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

I think/hope I addressed all points @echoix

Comment thread scripts/g.extension/g.extension.md Outdated
Comment on lines +266 to +343
Markus Neteler (original shell script)
Markus Neteler (original shell script)
Martin Landa, Czech Technical University in Prague, Czech Republic
(Python rewrite)
(Python rewrite)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this put the lines together?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wenzeslaus @echoix the automatic check suggest to remove the spaces, but it does indeed put the lines together. So keeping the spaces (as is was)?

Comment thread scripts/g.extension/g.extension.py Outdated
- Changed wording in code comment to reflect better that this new option only works for GRASS on MS-Windows
- added back double space after each author so they end up on a separate line.
Comment thread scripts/g.extension/g.extension.py Outdated
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs general HTML Related code is in HTML markdown Related to markdown, markdown files module Python Related code is in Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants