CI: switch to OSGeo4W v2 on MS Windows by ninsbl · Pull Request #1873 · OSGeo/grass

ninsbl · 2021-09-16T20:02:42Z

This PR replaces the build part of #1683
It will also change how OSGeo4W packages are build and use that same configuration for tests.
I leave the milestone open, as I do not think it should delay 8.0, even if it would be good to have testing with current OSGeo4W in place before releasing...

hellik · 2021-09-16T20:42:33Z

@ninsbl with a quick view on the PR, great work! I'll have a closer look in the next days.

@landam I think this PR is a very good starting point to unify CI and OSGeo4W build and packaging.

as 32bit is obsolete now with OSGeo4W, simplification of building and packaging can be done.

OSGeo4W v1 is no longer maintained anymore. how long do you consider to provide winGRASS builds for it?

I suggest to drop it with the upcoming 7.8 amd 8 releases.

hellik · 2021-09-22T16:30:40Z

 # and create bin.x86_64-w64-mingw32\grass$ver.bat (run this batch file to start
 # GRASS GIS) and dist.x86_64-w64-mingw32\etc\env.bat.
 #
-# -p	optionally install GRASS GIS to C:\OSGeo4W64\opt\grass (run


IIRC @HuidaeCho used -p to have zippable binaries.

Yes, I used -p to create an unzippable package, which is needed in a non-admin environment. Most of this original workflow script is now moved to mswindows/osgeo4w/build_osgeo4w.sh. What happened to the old OSGeo4W build script that was in mswindows/osgeo4w?

Yes, I used -p to create an unzippable package, which is needed in a non-admin environment.

Thanks for the info. I wasn`t aware of that. Will see that this is preserved...

Most of this original workflow script is now moved to mswindows/osgeo4w/build_osgeo4w.sh. What happened to the old OSGeo4W build script that was in mswindows/osgeo4w?

There is still package.sh in mswindows/osgeo4w. I think, in future it would be good to have one build recipe that is used for both packaging and testing (and thus lives in ./mswindows/ and not .github/workflows/ (unless we have a packaging workflow)). The build parts could then be removed from package.sh and just called...

HuidaeCho · 2021-09-22T19:15:03Z

 # and create bin.x86_64-w64-mingw32\grass$ver.bat (run this batch file to start
 # GRASS GIS) and dist.x86_64-w64-mingw32\etc\env.bat.
 #
-# -p	optionally install GRASS GIS to C:\OSGeo4W64\opt\grass (run


Yes, I used -p to create an unzippable package, which is needed in a non-admin environment. Most of this original workflow script is now moved to mswindows/osgeo4w/build_osgeo4w.sh. What happened to the old OSGeo4W build script that was in mswindows/osgeo4w?

HuidaeCho · 2021-09-22T19:20:03Z

+cp -a $dist $grass_path
+# have a versionless and versioned startup script
+cp -a $bin/grass.py $bin/grass$ver.py
+cp -a $bin/grass$ver.py $grass_path/etc


Do we still keep $ver in this case? In fact, I tried to rename it to simply $grass_path/etc/grass.py and when it was called from $grass_path/grass.bat, it was not able to import grass.grassdb.checks saying grass is not a package. I think that's because of grass.py trying to import grass.* from $grass_path/etc/python/grass/* when its name itself is grass (importing itself instead of $grass_path/etc/python/grass/*).

Yes, this is a problem. We need to resolve it some point.

HuidaeCho · 2021-09-22T19:22:09Z

+# and create bin.x86_64-w64-mingw32\grass$ver.bat (run this batch file to start
+# GRASS GIS) and dist.x86_64-w64-mingw32\etc\env.bat.
+#
+# -p	optionally install GRASS GIS to C:\OSGeo4W\opt\grass (run


-p is not moved here. It would still be good to have this option to create an unzippable package for non-admin users.

Imagery: move signature files to a <name>/<element> type directory structure (a.k.a. _misc) Full path to signature file now is: location/mapset/signatures/<type>/<name>/<element> This new structure will allow to store arbitrary number of files for each signature. None of existing signature types utilizes the new functionality.

* Added warning about region not being respected * break lines at approx 70-80 chars Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com> Co-authored-by: Veronica Andreo <veroandreo@gmail.com> Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com>

…SGeo#1866) Bandrefs: make band references optional to simplify generation of signature files. This brings back pre GRASS 8 like functionality by allowing to classify only the same imagery group if band references are missing.

…eo#1866 (OSGeo#1887)

* mswindows: Update crosscompile.sh to 8.0 * mswindows: Drop more versions * Python 38 => 39 * Keep $version in etc/grass$version.py to avoid import conflicts

… 4.1.1 (OSGeo#1890)

lib/manage: use correct buffer sizes and check for truncated strings affects mainly `g.rename`

By default rendering in jupyter notebook is based on the first layer extent and resolution. Saved region and current region can also be specified.

ninsbl · 2022-01-05T08:02:36Z

Rebase or merge with main won't harm, but if I understand correctly, the tests for a PR (really for PR, not for a push to your branch) run against a merged state (as if the PR branch would be merged to the base branch right now). In other words, rebase should not be needed for the PR tests, but to avoid any confusion, esp. when actions are changed, and to get the right state locally, I prefer to do rebase/merge need anyway.

I see. Trying a rebase out of a lack of better ideas.

I hope running it locally will tell you more.

Unfortunately, as this is MS Windows, I am not able to run it locally...
We`ll see whar rebase/merge does and then try something else if needed....

wenzeslaus · 2022-01-05T13:45:00Z

My only idea right now is to remove the version printing for this PR and resolve it later if needed.

nilason · 2022-01-05T18:05:39Z

This got me very curious, so I forked and tested on CI.

with test_simple.sh:

echo $PATH
/c/OSGeo4W/bin:/usr/bin:/mingw64/bin:/d/a/grass/grass/dist.x86_64-w64-mingw32/bin:/d/a/grass/grass/dist.x86_64-w64-mingw32/bash_bin

which grass
/d/a/grass/grass/dist.x86_64-w64-mingw32/bash_bin/grass

with print_versions.sh:

echo $PATH
/c/Program Files/PowerShell/7:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.3/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.1/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.2/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/c/Program Files/Git/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.15.15/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.312-7/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.4/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files/Git/cmd:/c/Program Files/Git/mingw64/bin:/c/Program Files/Git/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/runneradmin/.dotnet/tools:/c/Users/runneradmin/.cargo/bin:/c/Users/runneradmin/AppData/Local/Microsoft/WindowsApps

which grass
      0 [main] which (6380) C:\Program Files\Git\usr\bin\which.exe: *** fatal error - cygheap base mismatch detected - 0x180348408/0x180347408.

wenzeslaus · 2022-01-05T20:48:24Z

The diff includes load of old changes. The old commits still seem to be on the branch. Back up what you need (e.g. by creating another branch from you branch) and then try to move to the old commits. Best solution likely is to checkout the last commit which is okay (my guess is 61c7b29 or somewhere around), create a new branch, and then cherry-pick what you need from other commits (your unique commits). Then force-push the new branch to this remote branch (see "pushing local branch to a remote branch with a different name").

ninsbl · 2022-01-05T20:53:06Z

The diff includes load of old changes. The old commits still seem to be on the branch. Back up what you need (e.g. by creating another branch from you branch) and then try to move to the old commits. Best solution likely is to checkout the last commit which is okay (my guess is 61c7b29 or somewhere around), create a new branch, and then cherry-pick what you need from other commits (your unique commits). Then force-push the new branch to this remote branch (see "pushing local branch to a remote branch with a different name").

Oh, yes. What a mess, sorry. Thanks for the heads up!

wenzeslaus · 2022-01-05T21:00:43Z

My guess is that you rebased your local branch on top the main branch and then merged with your remote branch instead of replacing it with force push.

ninsbl · 2022-01-07T09:21:42Z

With still limited understanding of git, a quick and dirty atempt to cleanup the mess was to move the changes in a new, frech branch. Is it acceptable to just open a new PR from ninsbl@f7eb74d and replace this one? Again, sorry for the mess...
Printing the versions still fails (and I have no idea why) so I left it out for now...

wenzeslaus · 2022-01-07T14:27:25Z

...move the changes in a new, fresh branch. Is it acceptable to just open a new PR from ninsbl@f7eb74d and replace this one?

Yes, although replacing old remote with new local branch is simple: git push origin +new-local-branch:old-remote-branch (assuming origin points to your fork). The branch named old-remote-branch will still exist locally (until you do git branch -D old-remote-branch), so the risk involved is low (assuming you didn't do any changes to the branch online or pushed there from elsewhere).

A new PR might be good in anyway at this point just to get rid of the old discussion and start fresh as if nothing happened here.

Again, sorry for the mess...

PRs allow us to make mess, so that's perfectly fine.

Printing the versions still fails (and I have no idea why) so I left it out for now...

+1

wenzeslaus · 2022-01-07T14:44:23Z

Please, write a descriptive commit message for the new PR when merging. The description in the new PR links this one which links another one without providing much description on its own.

With this PR merged, CI uses packages from OSGeo4W Version2 (https://github.com/jef-n/OSGeo4W/) to build GRASS GIS using MSYS. Now also shellscript tests are executed. The PR replaces #1873 and parts of #1683. Compilation for tests is now also configured with OpenMP and LAPACK support (amongst others).

ninsbl · 2022-01-11T20:02:26Z

Replaced by #2074

With this PR merged, CI uses packages from OSGeo4W Version2 (https://github.com/jef-n/OSGeo4W/) to build GRASS GIS using MSYS. Now also shellscript tests are executed. The PR replaces OSGeo#1873 and parts of OSGeo#1683. Compilation for tests is now also configured with OpenMP and LAPACK support (amongst others).

ninsbl and others added 12 commits August 23, 2021 21:50

test binary and python

7664c7e

have a central build script

d8418aa

use central build script

d02348f

use OSGeo4W-NG path

3dd6ece

use OSGeo4W-NG setup

2116c22

py3 is now default

9d9a6b6

remove -p flag

fd0ec7d

set also PYTHONHOME and PYTHONUTF8

df1e310

add lapack to MSYS2

834066a

add various optional dependencies

61c7b29

Merge branch 'osgeo4w_ng' of github.com:ninsbl/grass into osgeo4w_ng

e39cb1a

no PDAL for now

dde9939

ninsbl added the CI Continuous integration label Sep 16, 2021

ninsbl requested review from hellik, landam and wenzeslaus September 16, 2021 20:02

wenzeslaus reviewed Sep 16, 2021

View reviewed changes

Comment thread .github/workflows/test_simple.sh Outdated

Comment thread .github/workflows/test_simple.sh Outdated

Comment thread .github/workflows/test_simple.sh

Comment thread .github/workflows/test_thorough.bat Outdated

configure: upgrade to autoconf 2.69 (OSGeo#1845)

dca9364

hellik reviewed Sep 22, 2021

View reviewed changes

HuidaeCho reviewed Sep 22, 2021

View reviewed changes

marisn and others added 9 commits September 23, 2021 11:07

add text snippet and example to add band references - edited from OSG…

f6487aa

…eo#1866 (OSGeo#1887)

v.out.ogr: Remove a garbage character from warning (OSGeo#1886)

1edc836

mswindows: Update crosscompile.sh to 8.0 (OSGeo#1876)

38c084a

* mswindows: Update crosscompile.sh to 8.0 * mswindows: Drop more versions * Python 38 => 39 * Keep $version in etc/grass$version.py to avoid import conflicts

wxGUI/gui_core: fix 'Set image size' dialog rendered size on wxPython…

e03efc2

… 4.1.1 (OSGeo#1890)

lib/manage: avoid buffer overflow (OSGeo#1899)

c522160

lib/manage: use correct buffer sizes and check for truncated strings affects mainly `g.rename`

grass.jupyter: more flexible region handling for rendering (OSGeo#1871)

2d2eff5

By default rendering in jupyter notebook is based on the first layer extent and resolution. Saved region and current region can also be specified.

ninsbl and others added 10 commits January 5, 2022 08:46

set also PYTHONHOME and PYTHONUTF8

0e4ffba

add lapack to MSYS2

4109b1a

add various optional dependencies

d8f2503

no PDAL for now

edabdaa

adress code review

1e775fb

use print_version.sh

3753f40

parametrize python

4a700c0

parametrize python

77a0f1a

print version from bash

750962f

Merge branch 'osgeo4w_v2' of github.com:ninsbl/grass into osgeo4w_v2

9caa772

no version printing for now

47d58b3

ninsbl pushed a commit to ninsbl/grass that referenced this pull request Jan 5, 2022

move from OSGeo#1873

ed27e36

ninsbl pushed a commit to ninsbl/grass that referenced this pull request Jan 5, 2022

move more from OSGeo#1873

f7eb74d

ninsbl mentioned this pull request Jan 7, 2022

CI: Switch to OSGeo4W V2 #2074

Merged

ninsbl closed this Jan 11, 2022

Conversation

ninsbl commented Sep 16, 2021

Uh oh!

hellik commented Sep 16, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hellik Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

HuidaeCho Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

ninsbl Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

HuidaeCho Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

HuidaeCho Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

wenzeslaus Jan 4, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuidaeCho Sep 22, 2021

Choose a reason for hiding this comment

Uh oh!

ninsbl commented Jan 5, 2022

Uh oh!

wenzeslaus commented Jan 5, 2022

Uh oh!

nilason commented Jan 5, 2022

Uh oh!

wenzeslaus commented Jan 5, 2022

Uh oh!

ninsbl commented Jan 5, 2022

Uh oh!

wenzeslaus commented Jan 5, 2022

Uh oh!

ninsbl commented Jan 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wenzeslaus commented Jan 7, 2022

Uh oh!

wenzeslaus commented Jan 7, 2022

Uh oh!

ninsbl commented Jan 11, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ninsbl commented Jan 7, 2022 •

edited

Loading