Repository navigation
Use the NPZ object serialization of Sigima in the Web API - #430
Merged
Merged
Conversation
The NPZ format of the Web API was implemented both here and in DataLab-Kernel, by copies that diverged. Sigima now provides the implementation (sigima.io.serialize_object_to_npz and deserialize_object_from_npz, DataLab-Platform/Sigima#90), taken from this module: the format and its behavior are unchanged. datalab.webapi.serialization now re-exports these functions for the routes and keeps only object_to_metadata (and its scalar conversion), specific to the API. The tests of the removed private helpers are covered by Sigima; the end-to-end serialization tests are kept. The Web API guide says where the format is implemented. Closes #429 Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #429 — companion of DataLab-Platform/Sigima#91 (same branch name, so that the CI tests both together).
Summary
The NPZ format of the Web API is now implemented once, by Sigima (
sigima.io.serialize_object_to_npz/deserialize_object_from_npz, taken from this module), instead of two diverging copies in DataLab and DataLab-Kernel.datalab/webapi/serialization.pyre-exports the Sigima functions (used byroutes.pyand the tests) and keeps onlyobject_to_metadataand its scalar conversion, specific to the API (−327 lines). The format and its behavior are unchanged.webapi_extra_test.py: the tests of the removed private helpers (_make_json_serializable,_deserialize_obj_metadata) are dropped, their cases being covered bysigima/tests/io/npz_unit_test.py; the end-to-end serialization tests (compression, errors, metadata with NumPy values, uniform and non-uniform coordinates, archives of earlier versions) are kept.Tests
datalab/tests/webapi_test.pyandwebapi_extra_test.pywith Sigimafeature/shared-npz-serialization: 62 passed.