Major rework based on Sympy and TEMGYM Core - #64
Merged
Merged
Conversation
Basics already work; currently adding a ton of tests (WIP)
for more information, see https://pre-commit.ci
Simulate point charge displacement vector field in detector coordinates with model and validate by guessing CoM parameters with `libertem.udf.com.guess_corrections()`
See LiberTEM/LiberTEM#1775 for discussion! * Undo sign change between column and pixel y coordinates since that was ineffective. * Add tests to confirm correct behavior of the model * Add additional tests for CoM consistency * Make inverted sign the expected behavior
Some functions in TemGymCore rely on `PixelsXY` and `CoordsYX` being arrays, so we use newly introduced single types here
* Extract mapping matrix from model * Fast forward simulation with that matrix * Tests WIP
This allows to better simulate a real scan shift, rotation and scale where the scan changes relative to the object. Previously, the scan coordinaates vs. simulated object coordinates were fixed and always identical. Fix handling of semiconv. Introduce many tests, incl descan error moving aperture outlines.
Enable float64 everywhere and manually specify inverses for good measure
WIP, flake8 skipped
* Tests for correction and back-projetion * Adapt OverfocusUDF to work again FIXME: How to update the parameters of an UDF without changing it? Important to keep plots alive!
* Update UDF to use new functions * Adapt notebooks to new parameters * Update tests * Simulation now includes semiconv, will be important for descan error optimization
The solver converges extremely quickly
The idea is a series of calibration functions to obtain a fully calibrated parameter set from a number of test measurements. * Wrapper function to run optax.lbfgs, which works EXTREMELY well on the model. * High level functions that take meaningful input parameters and starting parameters, and return updated parameters. TODO perhaps include the residual in the return value? * Descan error is WIP, will be based on measurements at different camera lengths. * Overfocus, rotation and flip TBD, in particular relative to descan error.
One will first have to compensate descan error, and then adjust other parameters such as flip_y, rotations etc. This change starts adding functions that allow setting these parameters while keeping the effect of descan error compensation wrt scan and detector coordinates the same. The transformations should be very fast for interactive use. For that reason they are implemented directly instead of using the solver method. TODO complete and add tests.
The `adjust_*()` methods allow first compensating the descan error based on incorrect model parameters, and then adjusting other model parameters while keeping the descan error compensated, i.e. they change the descan parameters in such a way that their effect relative to scan and detector pixel coordinates remains the same. This is helpful since e.g. calibrating camera length based on known diffraction angles may need descan compensation if position-averaged diffraction patterns from a 4D STEM dataset are to be used, and coordinate system calibration for scan step, scan rotation etc is easier as well if descan error is removed first. These methods are implemented as direct transformations rather than using a solver since they need to be fast enough for immediate response in interactive use.
Also adding a trace() function to reduce boilerplate since we are just interested in the tracing result from parameters in most cases. Refactoring all tests WIP
Works in principle, but can confuse downstream users. I guess it could be better to try preserve the original type of the input parameter to the optimization routine, but later(TM).
Descan error correction for single camera length works!
* Filter out reflections with low intensity from twothetas list * Use adjustment instead of deriving in optimization routines to maintain descan error compensation * Both point list and sharpness optimize detector rotation to keep scan rotation as a user-provided parameter
The large excitation error included weird duplicates Also, consider that diffsims calculates theta, not twotheta Add test case to check against literature values
* Add display of scale bar angle * Add display of all model parameters * Default model moved to Model4DSTEM since generally useful
* `CalibratedDataset` is a combination of dataset with model * Diff operation for calibrated DM4 dataset * Serialization and deserialization for calibrated DM4 dataset to allow checkpointing for complex analyses
* Correct Z direction (I assume longer CL when increasing Z means moving up) * Introduce function and test to invert the overfocus to obtain the equivalent mapping from specimen to detector
At this point no reason to be backwards compatible IMO.
uellue
added a commit
to uellue/TemGymCore
that referenced
this pull request
Oct 1, 2026
Chicken egg situation with LiberTEM/Microscope-Calibration#64 on the way. Tests pass locally with this current version
TODO how to make the model and its parameters very lightweight while also allowing to install all the heavy GUI etc?
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #64 +/- ##
===========================================
- Coverage 100.00% 94.79% -5.21%
===========================================
Files 5 9 +4
Lines 190 1442 +1252
Branches 18 108 +90
===========================================
+ Hits 190 1367 +1177
- Misses 0 49 +49
- Partials 0 26 +26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Code coverage issue from two missing bits:
|
for more information, see https://pre-commit.ci
Jax requires setting the float64 config early, and it is unclear how early is early enough... To avoid complication between Flake8 and Ruff, migrate everything to Ruff, which seems to support a superset of flake8. * Migrate pre-commit to ruff, sunset flake8 and remove its config * Also format with `ruff`, should make life easier in particular with agentic coding... * Migrate `flake8` config to ruff as far as possible * Apply new style everywhere
Makes life easier in IDEs -- mine seems to run both ruff and flake8...
Prominently, Widget.name -> Widget.label
This should boost coverage for stemoverfocus.py
Gotta pump these coverage numbers up...
...and also exercise the merge_all copied from PickUDF
Member
Author
|
Jax.jit was not harmful for coverage, it turns out. |
Member
Author
|
I'll just merge since this is essentially a rewrite of alpha code. We can rather review main... |
This was referenced Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds in large parts on #54