Repository navigation
Does mhkit work with the Nortek Aquadopp? #458
Description
Activity
@ddileonardo Looks like I forgot to update the reader docstring... It does support Aquadopps, but I've only had test data for the profiling versions. For those it's the
.prffile, for the single point meters it is the.aqdfiletype.Thanks! I have a .prf file. The instrument was set to profile.
I am getting a KeyError when I try to use dolfyn.read.
from mhkit import dolfyn data_path = r'GS-LMR_CH2ADP_020526.prf' ds = dolfyn.read(data_path)
KeyError Traceback (most recent call last) Cell In[3], line 2 1 data_path = r'GS-LMR_CH2ADP_020526.prf' ----> 2 ds = dolfyn.read(data_path) File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\api.py:114, in read(fname, userdata, nens, **kwargs) 112 func_map = dict(RDI=read_rdi, nortek=read_nortek, signature=read_signature) 113 func = func_map[file_type] --> 114 return func(fname, userdata=userdata, nens=nens, **kwargs) File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\nortek.py:60, in read_nortek(filename, userdata, debug, do_checksum, nens, **kwargs) 51 logging.basicConfig( 52 filename=str(logfile), 53 filemode="w", 54 level=logging.NOTSET, 55 format="%(name)s - %(levelname)s - %(message)s", 56 ) 58 userdata = base._find_userdata(filename, userdata) ---> 60 rdr = _NortekReader(filename, debug=debug, do_checksum=do_checksum, nens=nens) 61 rdr.readfile() 62 rdr.cleanup() File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\nortek.py:230, in _NortekReader.__init__(self, fname, endian, debug, do_checksum, bufsize, nens) 228 self.config["config_type"] = "ADV" 229 # Initialize the instrument type: --> 230 self._inst = self.config.pop("config_type") 231 # This is the position after reading the 'hardware', 232 # 'head', and 'user' configuration. 233 pnow = self.pos KeyError: 'config_type'
@ddileonardo Can you rebuild your conda environment with the latest version of MHKiT we just released? Looks like you're running an older version.
If you open up your .prf file in a text editor, you will will see a few ascii characters that spell out the serial number at the very beginning of the file. MHKiT reads this serial number, which will start with "AQD" or "PRF" for Aquadopp profilers, to determine the correct reader to use. If yours doesn't start with either of those, please let me know.I used the method from the documentation to install
conda install -c conda-forge mhkitSince it doesn't specify a version, I would have though it should be the most recent version? I installed in late July/early August.
When I try to update with conda, the only package that updates is openssl. I still get the same error.
I have tried creating a whole new environment and installing mhkit in the same way. I'm still getting the same error.
Is there a different conda channel that I need to use?
As far as the top of prf file goes, when I open in text editor I see a couple bits of binary and the AQD with a serial number so that seems right.
Reacted by James McVeyOh, please use
pip install mhkitto install the full package, orpip install mhkit[dolfyn]for just the ADCP-related dependencies. We haven't yet updated the conda-forge package for the latest release.After the pip install I get an immediate import error for bottleneck when trying to import dolfyn or mhkit. Is there a specific python version I need or something else?
My install history in the environment goes:
- conda install pythong
- conda install pip
- pip install mhkit
- pip install ipykernel
So all the package version should be defined by pip.
pandas 3.0.5 pypi_0 pypi
numpy 2.5.1 pypi_0 pypiImportError Traceback (most recent call last) Cell In[1], [line 1](vscode-notebook-cell:?execution_count=1&line=1) ----> [1](vscode-notebook-cell:?execution_count=1&line=1) from mhkit import dolfyn 2 from mhkit.dolfyn.adp import api 3 import numpy as np 4 from matplotlib import pyplot as plt File ~\.conda\envs\qbay-pip\Lib\site-packages\mhkit\__init__.py:5 2 import importlib 4 # Register datetime converter for a matplotlib plotting methods ----> [5](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/mhkit/__init__.py:5) from pandas.plotting import register_matplotlib_converters as _rmc 7 _rmc() 9 # Use targeted warning configuration File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\__init__.py:58 55 # let init-time option registration happen 56 import pandas.core.config_init # pyright: ignore[reportUnusedImport] # noqa: F401 ---> [58](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/__init__.py:58) from pandas.core.api import ( 59 # dtype 60 ArrowDtype, 61 Int8Dtype, 62 Int16Dtype, 63 Int32Dtype, 64 Int64Dtype, 65 UInt8Dtype, 66 UInt16Dtype, 67 UInt32Dtype, 68 UInt64Dtype, 69 Float32Dtype, 70 Float64Dtype, 71 CategoricalDtype, 72 PeriodDtype, 73 IntervalDtype, 74 DatetimeTZDtype, 75 StringDtype, 76 BooleanDtype, 77 # missing 78 NA, 79 isna, 80 isnull, 81 notna, 82 notnull, 83 # indexes 84 Index, 85 CategoricalIndex, 86 RangeIndex, 87 MultiIndex, 88 IntervalIndex, 89 TimedeltaIndex, 90 DatetimeIndex, 91 PeriodIndex, 92 IndexSlice, 93 # tseries 94 NaT, 95 Period, 96 period_range, 97 Timedelta, 98 timedelta_range, 99 Timestamp, 100 date_range, 101 bdate_range, 102 Interval, 103 interval_range, 104 DateOffset, 105 # conversion 106 to_numeric, 107 to_datetime, 108 to_timedelta, 109 # misc 110 Flags, 111 Grouper, 112 factorize, 113 unique, 114 NamedAgg, 115 array, 116 Categorical, 117 set_eng_float_format, 118 Series, 119 DataFrame, 120 ) 121 from pandas.core.col import col 123 from pandas.core.dtypes.dtypes import SparseDtype File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\api.py:[27](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/api.py:27) 16 from pandas.core.dtypes.missing import ( 17 isna, 18 isnull, 19 notna, 20 notnull, 21 ) 23 from pandas.core.algorithms import ( 24 factorize, 25 unique, 26 ) ---> 27 from pandas.core.arrays import Categorical 28 from pandas.core.arrays.boolean import BooleanDtype 29 from pandas.core.arrays.floating import ( 30 Float32Dtype, 3[1](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/__init__.py:1) Float64Dtype, 32 ) File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\__init__.py:1 ----> 1 from pandas.core.arrays.arrow import ArrowExtensionArray 2 from pandas.core.arrays.base import ( 3 ExtensionArray, 4 ExtensionOpsMixin, 5 ExtensionScalarOpsMixin, 6 ) 7 from pandas.core.arrays.boolean import BooleanArray File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\arrow\__init__.py:[5](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/arrow/__init__.py:5) 1 from pandas.core.arrays.arrow.accessors import ( 2 ListAccessor, 3 StructAccessor, 4 ) ----> 5 from pandas.core.arrays.arrow.array import ArrowExtensionArray 7 __all__ = ["ArrowExtensionArray", "ListAccessor", "StructAccessor"] File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\arrow\array.py:78 73 from pandas.core.arrays._utils import to_numpy_dtype_inference 74 from pandas.core.arrays.base import ( 75 ExtensionArray, 76 ExtensionArraySupportsAnyAll, 77 ) ---> [78](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/arrow/array.py:78) from pandas.core.arrays.masked import BaseMaskedArray 79 from pandas.core.arrays.string_ import StringDtype 80 import pandas.core.common as com File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\masked.py:56 45 from pandas.core.dtypes.dtypes import ( 46 ArrowDtype, 47 BaseMaskedDtype, 48 ) 49 from pandas.core.dtypes.missing import ( 50 array_equivalent, 51 is_valid_na_for_dtype, 52 isna, 53 notna, 54 ) ---> [56](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/masked.py:56) from pandas.core import ( 57 algorithms as algos, 58 arraylike, 59 missing, 60 nanops, 61 ops, 62 ) 63 from pandas.core.algorithms import ( 64 factorize_array, 65 isin, (...) 68 take, 69 ) 70 from pandas.core.array_algos import ( 71 masked_accumulations, 72 masked_reductions, 73 ) File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\nanops.py:[54](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/nanops.py:54) 51 if TYPE_CHECKING: 52 from collections.abc import Callable ---> 54 bn = import_optional_dependency("bottleneck", errors="warn") 55 _BOTTLENECK_INSTALLED = bn is not None 56 _USE_BOTTLENECK = False File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\compat\_optional.py:173, in import_optional_dependency(name, extra, min_version, errors) 171 minimum_version = min_version if min_version is not None else VERSIONS.get(parent) 172 if minimum_version: --> [173](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/compat/_optional.py:173) version = get_version(module_to_get) 174 if version and Version(version) < Version(minimum_version): 175 msg = ( 176 f"Pandas requires version '{minimum_version}' or newer of '{parent}' " 177 f"(version '{version}' currently installed)." 178 ) File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\compat\_optional.py:80, in get_version(module) 77 version = getattr(module, "__version__", None) 79 if version is None: ---> [80](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/compat/_optional.py:80) raise ImportError(f"Can't determine version for {module.__name__}") 81 if module.__name__ == "psycopg2": 82 # psycopg2 appends " (dt dec pq3 ext lo64)" to it's version 83 version = version.split()[0] ImportError: Can't determine version for bottleneck
Huh, we haven't seen that pandas error before. What happens if you run
pip install bottleneck?Looking at my package list, it seems like bottleneck wasn't installed before, for some reason. Installing it as above fixed the import issue.
Now though I am getting an error related to the beginning of the file:
WARNING:root:Unrecognized identifier: 0x9d
The first few characters look a bit like this when opened in notepad++
¥�� AQD11219
It looks like a bit of binary and then those ascii characters.
We are using legacy aquadopps, which might be useful information if the aquadopp workflow was developed with the newer gen 2 aquadopps.
Noted, that should've been installed with the package.
Yes, that's the serial number. I've tested with an older instrument than yours (based on the serial number) and one of the new ones.
Can you run
ds = dolfyn.read(data_path, debug=True)and upload the resulting log file here? There is no 0x9d ID, looking at Nortek's integrator guideHere is the log file
GS-LMR_CH2ADP_050726.dolfyn.log
I had a peek in there and it suggested to me that there was an issue with the data file. I tried a different data file and it read successfully.
Thank you very much for all your help.
Is there an odd number of bins in the datafile with the issue? And an even number of bins in the datafile that ran successfully?
There is a note in the docs that states the former will include an extra "fill" byte, but we've never seen that to be true before. But it looks like that might be true for your case.
Yes, there are 5 cells listed in the hdr file. There are 10 cells list in the hdr file of the file that works.
Reacted by James McVeyWell this might be really easy. Can you pull PR #459 and try that?
You could also copy-paste the changes in https://github.com/MHKiT-Software/MHKiT-Python/pull/459/changes if that's faster for you
Reacted by ddileonardoI'm trying, but it's not quite working.
I've cloned the repo, but I couldn't figure out how to pull a PR, so I just made the changes in nortek.py. But when I tried to build it as in the documentation
pip install -e .["all"]
Something didn't quite work. I've got python 3.14, but that wasn't an issue before.
This is the pip error from the build
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for numcodecs
Building wheel for bitshuffle (pyproject.toml) ... error
error: subprocess-exited-with-error× Building wheel for bitshuffle (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
running bdist_wheel
running build
running build_py
creating build\lib.win-amd64-cpython-314\bitshuffle
copying bitshuffle_init_.py -> build\lib.win-amd64-cpython-314\bitshuffle
running build_ext################################# # Compiling with OpenMP support # ################################# building 'bitshuffle.ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for bitshuffle
Successfully built mhkit
Failed to build cartopy aiohttp numcodecs bitshuffle
error: failed-wheel-build-for-install× Failed to build installable wheels for some pyproject.toml based projects
╰─> cartopy, aiohttp, numcodecs, bitshuffleAlright, we got there. And it works!
The install of all the required packages is potentially a big mess and MHKit is not properly installed as a package, but I've got al the files with the edits and was able to import them. And my odd bin file now works
Reacted by James McVeyGreat! But yes, sounds like we need to update our installation page
Reacted by ddileonardoI think the install files might need a check over, but for testing that fix, it works
Should I expect that the pip install will be updated with this fix or should I keep going with the test environment I just created?
Thanks again for all your help!
Happy to help! Please continue with the test environment, it might take a minute (read: week or two) to release a hotfix. The conda-forge release will also take about that time
If you'd like to have a clean environment, "cd" into the "MHKiT-Python" folder and run, from an anaconda prompt,
conda activate base conda env create conda activate mhkit-env pip install -e ".[all]"That should install the whole package properly, and then in the future run your python code after activating the "mhkit-env" environment.
Reacted by ddileonardoThat still fails at the same place as before. Not sure what is going on there, but I will likely just go with what I was able to cobble together before, and check back in a couple of weeks. I'm likely to be putting this project down for a couple of weeks starting tomorrow anyway.
Building wheels for collected packages: mhkit, bitshuffle
Building editable for mhkit (pyproject.toml) ... done
Created wheel for mhkit: filename=mhkit-1.1.0-0.editable-py3-none-any.whl size=9137 sha256=66f742b6ea01fc0b6cddeb235df6cc09dc77016190cd3e108a4fe55fae01a6fa
Stored in directory: C:\Users\ddileonardo\AppData\Local\Temp\pip-ephem-wheel-cache-d7k69etx\wheels\ba\e4\a3\133e46a242694d82f1dd6da9ea21e938cca9c013d0b8f1396a
Building wheel for bitshuffle (pyproject.toml) ... error
error: subprocess-exited-with-error× Building wheel for bitshuffle (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
running bdist_wheel
running build
running build_py
creating build\lib.win-amd64-cpython-311\bitshuffle
copying bitshuffle_init_.py -> build\lib.win-amd64-cpython-311\bitshuffle
running build_ext################################# # Compiling with OpenMP support # ################################# building 'bitshuffle.ext' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for bitshuffle
Successfully built mhkit
Failed to build bitshuffle
error: failed-wheel-build-for-install× Failed to build installable wheels for some pyproject.toml based projects
╰─> bitshuffleThis looks like some sort of windows installation problem with python or certain python packages. It sounds like the conda-forge version installed fine for you, and we'll have that out soon.
@jmcvey3 and @ddileonardo,
We just updated our experimental desktop application to mhkit 1.1.1 which leverages dolfyn "under the hood":
https://github.com/MHKiT-Software/mhkit-dolfyn-gui
There is a windows executable here: https://github.com/MHKiT-Software/mhkit-dolfyn-gui/releases/tag/v0.3.2
That has the python libraries "inside" and should work for reading Nortek Aquadopp files, but please let us know if that does not work.
I'm also working on the conda-forge release which has some non dolfyn related issues and can report back when that is available
@ddileonardo to get the latest mhkit dolfyn (v1.1.1) within a conda environment the steps below should work:
conda env create mhkit-env --python=3.11conda activate mhkit-envpip install "mhkit[dolfyn]"Checking the mhkit version:
python -c "import mhkit; print(mhkit.__version__)"Should yield:
v1.1.1
Note:
pip install mhkitdoesn't install any libraries, which is why you get import issues (this is a bit confusing, we should probably make this more obvious). You have to specify a module which is detailed in the documentation here https://github.com/MHKiT-Software/MHKiT-Python#module-specific-install-using-pip.And if you want all modules in mhkit pass
all, ie `pip install "mhkit[all]".
Does mhkit work with the Nortek Aquadopp? Specifically, I have data from the legacy (pre-2024) aquadopp. It is an ADCP, but it isn't mentioned in the documentation anywhere.
If mhkit does support aquadopps, which output file extension should I be trying to read?