fix: update runner.sh and endoscopic notebook for MONAI 1.6 tutorial runner compatibility#2065
fix: update runner.sh and endoscopic notebook for MONAI 1.6 tutorial runner compatibility#2065garciadias wants to merge 12 commits into
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughRunner papermill controls updated to exempt two notebooks and skip one; a spleen notebook updates its PyTorch Lightning pin and fixes a JSON outputs block; several notebooks have cosmetic f-string spacing edits; a detailed diagnostics report document is added. ChangesNotebook Processing Configuration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@runner.sh`:
- Around line 87-88: The doesnt_contain_max_epochs array entry is incorrect
because hovernet_infer_compare.ipynb does not exist; update the array (variable
doesnt_contain_max_epochs) to reference the actual notebook that lacks
max_epochs — replace hovernet_infer_compare.ipynb with
monailabel/monailabel_pathology_HoVerNet_QuPath.ipynb (or remove the nonexistent
entry) so the basename comparison ([[ "$e" == "$filename" ]]) can correctly
match and exempt the HoverNet inference notebook; ensure the
msd_crossval_datalist_generator.ipynb entry remains unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 790edfde-41e1-40f1-9d11-a3eb3d3813d6
📒 Files selected for processing (2)
computer_assisted_intervention/endoscopic_inbody_classification.ipynbrunner.sh
👮 Files not reviewed due to content moderation or server errors (1)
- computer_assisted_intervention/endoscopic_inbody_classification.ipynb
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
bundle/05_spleen_segmentation_lightning.ipynb (1)
41-41: ⚡ Quick winCritical fix for Python 3.12 mlflow import chain failure — well documented.
This change resolves the issue where pytorch-lightning 2.0.x eagerly imports mlflow at module level, causing failures under Python 3.12. The
>=2.1constraint ensures lazy mlflow imports are used.Optional: Consider a more conservative version constraint
The current constraint
>=2.1permits any future major version (3.x, 4.x, etc.), which could introduce breaking changes. For better stability, consider:-!pip install -q pytorch-lightning>=2.1 +!pip install -q "pytorch-lightning>=2.1,<3"However, since this is tutorial code and has been tested with 2.6.5, the current permissive constraint is acceptable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@bundle/05_spleen_segmentation_lightning.ipynb` at line 41, Replace the notebook installation line string "!pip install -q pytorch-lightning>=2.1" with a constrained version to avoid Python 3.12 mlflow import issues; update the command in the cell that currently contains "!pip install -q pytorch-lightning>=2.1" to either "!pip install -q 'pytorch-lightning>=2.1,<3.0'" for a conservative pin or keep ">=2.1" if you accept the permissive range, ensuring the cell is updated accordingly.diagnose_1_6_release.md (2)
259-263: 💤 Low valueAdd language identifier to fenced code block.
The error message block should specify a language for proper syntax highlighting and markdown validation.
📝 Suggested fix
`mlflow 3.13.0` (installed in the image) fails on Python 3.12 with: -``` +```python ImportError: attempted relative import beyond top-level package</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@diagnose_1_6_release.mdaround lines 259 - 263, Update the fenced code block
that contains the ImportError message so it includes a language identifier;
replace the opening "" before the "ImportError: attempted relative import beyond top-level package" line with "python" (the block referencing
mlflow.utils.uv_utils and the relative import error) to enable proper syntax
highlighting and markdown validation.</details> <!-- cr-comment:v1:a0bf915188f7b1372abe3425 --> --- `54-66`: _⚡ Quick win_ **Minor documentation gap: second notebook not listed.** The PR objectives state that both `msd_crossval_datalist_generator.ipynb` and `hovernet_infer_compare.ipynb` were added to the `doesnt_contain_max_epochs` exemption list, but this Category 2 section only documents `msd_crossval_datalist_generator.ipynb`. Line 438 in the "Fixes Applied" section confirms both notebooks were added to the exemption list. Consider adding `hovernet_infer_compare.ipynb` to the table in this section for completeness, or add a note explaining why it's not listed here (e.g., if it didn't fail in the initial run). <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@diagnose_1_6_release.md` around lines 54 - 66, The Category 2 — MissingKeyword table is missing hovernet_infer_compare.ipynb even though the "Fixes Applied" (line ~438) shows both msd_crossval_datalist_generator.ipynb and hovernet_infer_compare.ipynb were added to the doesnt_contain_max_epochs exemption; update the Category 2 table to include hovernet_infer_compare.ipynb with the same detail entry (or add a clarifying note why it was omitted), and ensure the documentation references the runner.sh doesnt_contain_max_epochs exemption so readers can reconcile the table with the applied fixes. ``` </details> <!-- cr-comment:v1:7bda2fd71a0aac0f57c28c85 --> </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.Inline comments:
In@diagnose_1_6_release.md:
- Around line 444-450: Remove the duplicate "PEP8 in 3 notebooks" entry from the
"Still pending" section and ensure the "Recommended Next Steps" section keeps
the existing "PEP8 autofix — all three notebooks autofixed withrunner.sh --autofix. ✓ DONE" text; specifically delete the table row or bullet that
contains the exact phrase "PEP8 in 3 notebooks" (and any associated action text)
so the document no longer shows PEP8 fixes as both pending and done, keeping
references to the affected notebooks
(surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb,
deep_atlas/deep_atlas_tutorial.ipynb,
modules/interpretability/class_lung_lesion.ipynb) only in the completed note
if needed.
Nitpick comments:
In@bundle/05_spleen_segmentation_lightning.ipynb:
- Line 41: Replace the notebook installation line string "!pip install -q
pytorch-lightning>=2.1" with a constrained version to avoid Python 3.12 mlflow
import issues; update the command in the cell that currently contains "!pip
install -q pytorch-lightning>=2.1" to either "!pip install -q
'pytorch-lightning>=2.1,<3.0'" for a conservative pin or keep ">=2.1" if you
accept the permissive range, ensuring the cell is updated accordingly.In
@diagnose_1_6_release.md:
- Around line 259-263: Update the fenced code block that contains the
ImportError message so it includes a language identifier; replace the opening
"" before the "ImportError: attempted relative import beyond top-level package" line with "python" (the block referencing mlflow.utils.uv_utils and
the relative import error) to enable proper syntax highlighting and markdown
validation.- Around line 54-66: The Category 2 — MissingKeyword table is missing
hovernet_infer_compare.ipynb even though the "Fixes Applied" (line ~438) shows
both msd_crossval_datalist_generator.ipynb and hovernet_infer_compare.ipynb were
added to the doesnt_contain_max_epochs exemption; update the Category 2 table to
include hovernet_infer_compare.ipynb with the same detail entry (or add a
clarifying note why it was omitted), and ensure the documentation references the
runner.sh doesnt_contain_max_epochs exemption so readers can reconcile the table
with the applied fixes.</details> <details> <summary>🪄 Autofix (Beta)</summary> Fix all unresolved CodeRabbit comments on this PR: - [ ] <!-- {"checkboxId": "4b0d0e0a-96d7-4f10-b296-3a18ea78f0b9"} --> Push a commit to this branch (recommended) - [ ] <!-- {"checkboxId": "ff5b1114-7d8c-49e6-8ac1-43f82af23a33"} --> Create a new PR with the fixes </details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: Organization UI **Review profile**: CHILL **Plan**: Pro **Run ID**: `a3e94dda-8fe0-49e3-9b18-8009fe13d5e2` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 22cb7bc89c659084d8288f01d6c74391d2e057e6 and f2e31eadb78677e35c1bb004cc6de52a66aaf3a7. </details> <details> <summary>📒 Files selected for processing (5)</summary> * `bundle/05_spleen_segmentation_lightning.ipynb` * `competitions/MICCAI/surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb` * `deep_atlas/deep_atlas_tutorial.ipynb` * `diagnose_1_6_release.md` * `modules/interpretability/class_lung_lesion.ipynb` </details> <details> <summary>✅ Files skipped from review due to trivial changes (2)</summary> * competitions/MICCAI/surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb * deep_atlas/deep_atlas_tutorial.ipynb </details> <details> <summary>👮 Files not reviewed due to content moderation or server errors (1)</summary> * modules/interpretability/class_lung_lesion.ipynb </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
…lity Add msd_crossval_datalist_generator.ipynb and hovernet_infer_compare.ipynb to doesnt_contain_max_epochs (inference/datalist notebooks with no training loop). Skip image_restoration.ipynb until monai.networks.nets.Restormer is merged into the dev branch. Fix endoscopic_inbody_classification notebook: pass return_state_dict=False to monai.bundle.load so it returns an nn.Module rather than a raw OrderedDict, which caused AttributeError on .train() with MONAI 1.6 API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
for more information, see https://pre-commit.ci Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
- competitions/MICCAI/surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb: E225 - deep_atlas/deep_atlas_tutorial.ipynb: E225 - modules/interpretability/class_lung_lesion.ipynb: E231 in f-string indexing Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…g pin pytorch-lightning~=2.0.0 (2.0.9) imports mlflow at module level via pytorch_lightning.loggers.mlflow; mlflow 3.13.0 fails to initialize under Python 3.12, making `import pytorch_lightning` fail in the training subprocess. pytorch-lightning>=2.1 uses lazy mlflow imports; confirmed working with 2.6.5: training and evaluation run to completion. Also documents the root cause in diagnose_1_6_release.md: two-part R7 issue (fd-limit download truncation + mlflow/pl import chain). Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
- Revert return_state_dict=False from endoscopic_inbody_classification.ipynb:
the upstream MONAI dev (≥1.5) removed the deprecated param entirely and
load() now returns nn.Module by default. The extra kwarg caused a TypeError
in CI. The original notebook is correct for MONAI ≥1.5.
- Update diagnose_1_6_release.md:
- R5: clarify it is a local-only issue (our MONAI @ 19cab577 still has the
deprecated return_state_dict=True default; upstream dev does not)
- R8: mark confirmed fixed — isolated run with --ulimit nofile=65536:65536
completed all 39 cells in 3m1s with no ancdata error (2026-06-11)
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
garciadias
force-pushed
the
fix/monai_1_6_release_notebooks
branch
from
9f83ff4 to
f3f49ff
Compare
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
There was a problem hiding this comment.
♻️ Duplicate comments (1)
diagnose_1_6_release.md (1)
452-452:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winDuplicate finding: PEP8 entry should be removed from "Still pending" section.
Line 452 lists "PEP8 in 3 notebooks" as an action still needed, but lines 398 and 420 both mark PEP8 autofix as "✓ DONE". The review stack context confirms that PEP8 f-string spacing fixes were applied to all three notebooks in this PR.
This inconsistency was already flagged in a previous review comment.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@diagnose_1_6_release.md` at line 452, Remove the duplicate "PEP8 in 3 notebooks | Run `bash runner.sh --autofix` for `surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb`, `deep_atlas/deep_atlas_tutorial.ipynb`, `modules/interpretability/class_lung_lesion.ipynb`" row from the "Still pending" section in diagnose_1_6_release.md (the entry that lists "PEP8 in 3 notebooks"), since those notebooks are already marked "✓ DONE" elsewhere; keep the completed entries and ensure only the completed PEP8 autofix rows remain.
🧹 Nitpick comments (1)
diagnose_1_6_release.md (1)
259-261: 💤 Low valueAdd language specifier to fenced code block.
The fenced code block at line 259 is missing a language specifier. Consider adding
textorpythonto improve readability and satisfy Markdown linting rules.📝 Proposed fix
-``` +```text ImportError: attempted relative import beyond top-level package</details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@diagnose_1_6_release.mdaround lines 259 - 261, The fenced code block
containing "ImportError: attempted relative import beyond top-level package" is
missing a language specifier; update the opening fence fromtotext (orchange the opening fence to ```text and leave the content and closing fence unchanged).Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@diagnose_1_6_release.md`:
- Line 452: Remove the duplicate "PEP8 in 3 notebooks | Run `bash runner.sh
--autofix` for `surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb`,
`deep_atlas/deep_atlas_tutorial.ipynb`,
`modules/interpretability/class_lung_lesion.ipynb`" row from the "Still pending"
section in diagnose_1_6_release.md (the entry that lists "PEP8 in 3 notebooks"),
since those notebooks are already marked "✓ DONE" elsewhere; keep the completed
entries and ensure only the completed PEP8 autofix rows remain.
---
Nitpick comments:
In `@diagnose_1_6_release.md`:
- Around line 259-261: The fenced code block containing "ImportError: attempted
relative import beyond top-level package" is missing a language specifier;
update the opening fence from ``` to ```text (or ```python if preferred) so the
block becomes a labeled fenced code block (e.g., change the opening fence to
```text and leave the content and closing fence unchanged).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0bb72274-f31b-4261-9885-6fd6814a3de1
📒 Files selected for processing (7)
bundle/05_spleen_segmentation_lightning.ipynbcompetitions/MICCAI/surgtoolloc/preprocess_detect_scene_and_split_fold.ipynbcomputer_assisted_intervention/endoscopic_inbody_classification.ipynbdeep_atlas/deep_atlas_tutorial.ipynbdiagnose_1_6_release.mdmodules/interpretability/class_lung_lesion.ipynbrunner.sh
✅ Files skipped from review due to trivial changes (1)
- deep_atlas/deep_atlas_tutorial.ipynb
🚧 Files skipped from review as they are similar to previous changes (3)
- runner.sh
- bundle/05_spleen_segmentation_lightning.ipynb
- competitions/MICCAI/surgtoolloc/preprocess_detect_scene_and_split_fold.ipynb
👮 Files not reviewed due to content moderation or server errors (2)
- modules/interpretability/class_lung_lesion.ipynb
- computer_assisted_intervention/endoscopic_inbody_classification.ipynb
The notebook calls .to(device) where device is CUDA unconditionally; no CPU fallback exists. Fails with AssertionError: Torch not compiled with CUDA enabled on the CPU-only GitHub Actions runner. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
deep_atlas_tutorial.ipynb hardcodes device = torch.device("cuda:0") with no
CPU fallback; fails with AssertionError: Torch not compiled with CUDA enabled
on the CPU-only GitHub Actions runner.
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…ax_epochs; fix stale pending table - runner.sh: hovernet_infer_compare.ipynb does not exist in the repo; the exemption was a dead entry that could never match. Removed. - diagnose_1_6_release.md: PEP8 row removed from the 'Still pending' table (autofix was already applied and marked done at line 401). Also updated runner.sh change summary to reflect the actual skip list entries (deep_atlas_tutorial, 05_spleen_segmentation_lightning). Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…r.sh
--jobs N (default 1, backwards-compatible):
Runs N notebooks concurrently via background subshells + wait -n
semaphore. Per-notebook stdout/stderr goes to an isolated temp log;
logs are printed in original notebook order after all jobs finish.
Recommended value for download-heavy runs: 4-8. Use --jobs 1 when
notebooks do conflicting pip installs (e.g. GPU-training notebooks).
--data-dir PATH:
Exports MONAI_DATA_DIRECTORY=PATH before notebooks run. 109/117
runnable notebooks honour this env var and fall back to a temp dir
when it is absent -- so without it every container run re-downloads
all datasets. Setting it to a bind-mounted host path caches data
permanently.
Typical Docker invocation:
docker run ... \
-v /host/monai_data:/data/monai_cache \
monai_1_6:latest \
bash -c "cd /opt/tutorials && bash runner.sh --jobs 4 --data-dir /data/monai_cache"
Also respects MONAI_DATA_DIRECTORY already set in the environment
(e.g. passed via docker run -e) without requiring --data-dir.
Per-notebook logic extracted into _run_notebook() so both paths share
one implementation; sequential path behaviour is unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
garciadias
force-pushed
the
fix/monai_1_6_release_notebooks
branch
from
d599eb7 to
1a760d6
Compare
|
Hi @garciadias thanks for this huge effort in going through the tutorials thoroughly. Is there remedial work to do still? We merged the PR on core for the Dockerfile updates, does that now address any issue here? I know @vikashg was also looking into the tutorials, so we have thorough testing between you and can reconcile PRs. |
I think all is done here. If I understand correctly, what @vikashg was proposing was to have an agentic system to self-correct the tutorials in the future, but I think this can be done independently of this PR. |
- computer_assisted_intervention/endoscopic_inbody_classification.ipynb: revert output/execution_count stripping on the training cell; this was the only diff to the file and the cell's source was otherwise unchanged, per @ericspod's review comment
|
Per @ericspod's feedback, moving the full diagnostics report here instead of committing it to the repo as a standalone file. MONAI 1.6 Release — Tutorial Test DiagnosticsDate: 2026-06-10 Summary
Category 1 — PEP8 Violations (3 notebooks)The flake8 check (run via
Category 2 — MissingKeyword (1 notebook)The runner requires all training notebooks to declare
Fix options:
Category 3 — ExecutionError (91 notebooks)Papermill returned a non-zero exit code. Specific errors are in stderr (not captured in 3a — Generative model training (30 notebooks)These notebooks train diffusion/VAE/GAN models. Even with
3b — 3D segmentation / AutoSeg training (6 notebooks)These notebooks train segmentation models (Spleen, BraTS, VISTA-3D). They need the
3c — External service / tool dependency (7 notebooks)These notebooks require external services (tracking servers, serving frameworks, or desktop
3d — MONAI core modules / data loading (48 notebooks)These are general-purpose tutorials covering transforms, networks, datasets, and workflows.
Environment NotesKey changes made to the Docker image to reach this test run (from baseline
Working docker run commanddocker --context default run --gpus all --rm \
--entrypoint bash \
-e NVIDIA_DISABLE_REQUIRE=true \
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --ulimit nofile=65536:65536 \
-v /data/rgd/tutorials:/opt/tutorials \
-v /data/rgd/MONAI:/opt/monai \
monai_1_6:latest \
-c "cd /opt/tutorials && bash runner.sh -t <notebook_path> 2>&1"
Rerun Results (stderr captured) — 2026-06-11A targeted rerun of the 80 "our-only" failing notebooks was performed with Outcome
All 30 generation notebooks passed (they download their own small synthetic datasets, The 15 remaining failuresGroup R1 — mlflow 3.13.0 broken on Python 3.12 (4 notebooks)
The error originates inside
Fix: pin RUN pip install "mlflow<3.0"Eric passes these notebooks — his environment likely has an older mlflow. Group R2 — Disk full
|
| Notebook | What it tried to write |
|---|---|
deep_atlas/deep_atlas_tutorial.ipynb |
OASIS dataset (~2 GB) |
deployment/bentoml/mednist_classifier_bentoml.ipynb |
MedNIST dataset |
experiment_management/bundle_integrate_mlflow.ipynb |
Spleen bundle run artefacts (ran 221 min before failing) |
microscopy/multichannel_microscopy_classification.ipynb |
Pre-trained DenseNet169 weights |
This is a run-order artifact, not a persistent issue on this host. The root
filesystem has 39 GB free and the 3.7 GB generation datasets are now cached. On
subsequent runs with the generation data already present these notebooks complete
normally — confirmed by rerun on 2026-06-11 (runner_output_rerun_r2r7r8.logs).
Note: bundle_integrate_mlflow also requires the mlflow fix (Group R1) to be in
the rebuilt image before it can complete.
Group R3 — transformers 5.10.2 + PyTorch nv25.03 incompatibility (1 notebook)
transformers 5.10.2 references torch.float8_e8m0fnu which does not exist in
PyTorch 2.7.0a0+nv25.03, causing import of PreTrainedModel to fail.
| Notebook | Error |
|---|---|
hugging_face/hugging_face_pipeline_for_monai.ipynb |
ModuleNotFoundError: Could not import module 'PreTrainedModel' (caused by AttributeError: module 'torch' has no attribute 'float8_e8m0fnu') |
Fix: Pin transformers<5.0 in constraint file, or wait for nv25 PyTorch update.
Eric's PyTorch 2.12+cu130 supports float8_e8m0fnu; ours does not.
Group R4 — Missing MONAI module (1 notebook)
| Notebook | Error |
|---|---|
2d_regression/image_restoration.ipynb |
ModuleNotFoundError: No module named 'monai.networks.nets.restormer' |
Restormer is not yet present in our MONAI dev branch (19cab577). Eric is on
eccefc57 (+143 commits) which may already include it, or the notebook needs updating.
Fix: Cherry-pick the Restormer commit into the dev branch, or add the
notebook to skip_run_papermill until the class is merged.
Group R5 — MONAI bundle.load API — local-only issue (1 notebook)
| Notebook | Error (local MONAI dev @ 19cab577 only) |
|---|---|
computer_assisted_intervention/endoscopic_inbody_classification.ipynb |
AttributeError: 'collections.OrderedDict' object has no attribute 'train' |
monai.bundle.load() in our local MONAI dev branch (19cab577) has
@deprecated_arg("return_state_dict", since="1.2", removed="1.5") with
return_state_dict=True still active as the default, so it returns an
OrderedDict instead of an nn.Module.
This is a local-environment-only issue. The upstream MONAI dev branch
(eccefc57, used by CI and the upstream Docker image) removed the
return_state_dict parameter in MONAI 1.5, and load() now returns nn.Module
by default. The notebook is correct as-is for any MONAI ≥ 1.5.
No notebook fix needed. The return_state_dict=False workaround that was
previously applied broke CI because the upstream MONAI does not accept that
parameter at all (TypeError: unexpected keyword argument). It has been reverted.
Group R6 — Missing aim package (1 notebook)
| Notebook | Error |
|---|---|
experiment_management/spleen_segmentation_aim.ipynb |
ModuleNotFoundError: No module named 'aim' |
Fix: Add aim to requirements-dev.txt and reinstall in the image.
Group R7 — pytorch-lightning → mlflow import chain failure (1 notebook)
| Notebook | Error |
|---|---|
bundle/05_spleen_segmentation_lightning.ipynb |
First run: ContentTooShortError (89 min download truncated). Rerun with --ulimit nofile=65536:65536: download succeeds in 6 min but training cell fails with OptionalImportError: from scripts.main import train (No module named 'pytorch_lightning'). |
Root cause chain:
- Notebook installs
pytorch-lightning~=2.0.0→ installs 2.0.9 pytorch_lightning2.0.x eagerly importsmlflowat module level (viapytorch_lightning.loggers.mlflow)- mlflow 3.13.0 fails to initialize under Python 3.12 (same R1 root cause)
- Result:
import pytorch_lightningfails in the%%bashtraining subprocess
Fix: Change !pip install -q pytorch-lightning~=2.0.0 → !pip install -q "pytorch-lightning>=2.1".
pytorch-lightning ≥ 2.1 uses lazy mlflow imports; tested with 2.6.5, training and evaluation pass.
The original download truncation (ContentTooShortError) was due to fd limits (R8) and is also resolved with --ulimit nofile=65536:65536.
Group R8 — Socket resource exhaustion (1 notebook)
| Notebook | Error |
|---|---|
modules/public_datasets.ipynb |
RuntimeError: received 0 items of ancdata |
PyTorch DataLoader failed to pass file descriptors through Unix sockets between
the main process and worker processes (ancillary data = file-descriptor passing).
This happens when the per-process open-file-descriptor limit is too low
(Docker default: 1024; DataLoader workers need ~65 k).
Fix: Add --ulimit nofile=65536:65536 to the docker run command (already
reflected in the working command above).
Rerun note: The targeted rerun (runner_output_rerun_r2r7r8.logs) ran public_datasets in
the same container as deep_atlas. That notebook pip-installs packages which upgraded urllib3
to 2.x; papermill then failed to import immediately (ModuleNotFoundError: No module named 'urllib3.packages.six.moves') — a run-order contamination, not the ancdata fix being tested.
Confirmed fixed in isolated run (container with --ulimit nofile=65536:65536, no shared
container): all 39 cells passed, real 3m1s. No ancdata error observed.
Group R9 — MissingKeyword (1 notebook)
| Notebook | Error |
|---|---|
auto3dseg/notebooks/msd_crossval_datalist_generator.ipynb |
max_epochs not found; not in exemption list |
Known issue (documented in Category 2 above). Add to doesnt_contain_max_epochs.
Recommended Next Steps
- Pin
mlflow<3.0in the Dockerfile — fixes 4 notebooks (Group R1). ✓ DONE (PR #8912) - R2 is a run-order artifact — not applicable on this host (39 GB free, data cached).
--ulimit nofile=65536:65536added to working docker run command — fixes R8. ✓ DONE (confirmed clean run 3m1s)- PEP8 autofix — all three notebooks autofixed with
runner.sh --autofix. ✓ DONE - Fix MissingKeyword —
msd_crossval_datalist_generator.ipynbadded to exemption list. ✓ DONE (PR fix: update runner.sh and endoscopic notebook for MONAI 1.6 tutorial runner compatibility #2065) - Pin
transformers<5.0— fixes R3. ✓ DONE (PR #8912) - Add
aimtorequirements-dev.txt— fixes R6. ✓ DONE (PR #8912) - R5 is local-only — no notebook fix needed; upstream MONAI ≥ 1.5 removed the deprecated param and
load()returnsnn.Moduledirectly. ✓ CONFIRMED (reverted wrong fix) - Skip
image_restoration.ipynbuntil Restormer is merged (R4). ✓ DONE (PR fix: update runner.sh and endoscopic notebook for MONAI 1.6 tutorial runner compatibility #2065) - Fix R7 —
pytorch-lightning>=2.1pin inbundle/05_spleen_segmentation_lightning.ipynb. ✓ DONE (PR fix: update runner.sh and endoscopic notebook for MONAI 1.6 tutorial runner compatibility #2065)
Priority order
| Priority | Action | Impact | Status |
|---|---|---|---|
| High | Pin mlflow<3.0 (Dockerfile rebuild) |
+4 passes | ✓ Done (PR #8912) |
| High | R2 disk-full | +4 passes | ✓ Not an issue (run-order artifact) |
| Medium | --ulimit nofile=65536:65536 in docker run |
+1 pass | ✓ Done (confirmed, 3m1s clean run) |
| Medium | Add msd_crossval_datalist_generator to exemption |
+1 pass | ✓ Done (PR #2065) |
| Medium | Pin transformers<5.0 |
+1 pass | ✓ Done (PR #8912) |
| Low | Add aim to requirements |
+1 pass | ✓ Done (PR #8912) |
| Low | R5 bundle.load API — local-only, no fix needed |
+1 pass (upstream) | ✓ Confirmed (reverted bad fix) |
| Low | PEP8 autofix (3 notebooks) | 0 fails eliminated | ✓ Done |
| Low | Skip image_restoration.ipynb (Restormer missing) |
+1 pass | ✓ Done (PR #2065) |
| Low | Fix Spleen Lightning notebook (pytorch-lightning>=2.1) |
+1 pass | ✓ Done (PR #2065) |
Fixes Applied (2026-06-11)
Changes committed to bring Docker run to parity with Eric's run:
MONAI repo (/data/rgd/MONAI)
| File | Change |
|---|---|
Dockerfile |
Base image: 24.10-py3 → 25.03-py3; rebuild pip constraint file (keep numpy==1.26.4, add setuptools<71); install papermill jupytext autopep8 autoflake ipywidgets; pin mlflow<3.0, transformers<5.0; add aim, lightning>=2.0 |
requirements-dev.txt |
Pin transformers<5.0; pin mlflow<3.0; add aim; add lightning>=2.0; remove python_version<=3.10 caps from cucim, onnxruntime, transformers |
Tutorials repo (/data/rgd/tutorials)
| File | Change |
|---|---|
runner.sh |
Add msd_crossval_datalist_generator.ipynb to doesnt_contain_max_epochs; add image_restoration.ipynb, 05_spleen_segmentation_lightning.ipynb, deep_atlas_tutorial.ipynb to skip_run_papermill |
computer_assisted_intervention/endoscopic_inbody_classification.ipynb |
Reverted return_state_dict=False — upstream MONAI ≥1.5 already returns nn.Module by default; the extra kwarg caused TypeError in CI |
bundle/05_spleen_segmentation_lightning.ipynb |
Change pytorch-lightning~=2.0.0 → pytorch-lightning>=2.1 to avoid mlflow eager-import failure (R7) |
Still pending (require environment changes or separate PRs)
| Issue | Action needed |
|---|---|
| Disk space (R2) | Free /data disk or bind-mount scratch volume; deep_atlas (~2 GB), deployment/bentoml, experiment_management/bundle_integrate_mlflow, microscopy notebooks |
| Socket FD limit (R8) | ✓ Confirmed fixed — --ulimit nofile=65536:65536 resolves ancdata error; isolated run completed in 3m1s (2026-06-11) |
| pytorch-lightning/mlflow import chain (R7) | ✓ Fixed: pytorch-lightning>=2.1 in notebook (PR #2065) |
| Restormer in MONAI dev | Cherry-pick Restormer network class commit; remove from skip list once merged |
- computer_assisted_intervention/endoscopic_inbody_classification.ipynb: revert output/execution_count stripping on the training cell; this was the only diff to the file and the cell's source was otherwise unchanged, per @ericspod's review comment Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Full diagnostics report will be posted as a PR comment instead of committed to the repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
garciadias
force-pushed
the
fix/monai_1_6_release_notebooks
branch
from
2b4687e to
4422157
Compare
2 participants
Description
Fixes tutorial runner failures observed during MONAI 1.6 release testing (Docker image, Python 3.12, RTX 5090). Full diagnostics in this comment.
Failures addressed
mlflow 3.xfails on Python 3.12 (from .. import zipprelative import)mlflow<3.0inrequirements-dev.txttransformers 5.xreferencestorch.float8_e8m0fnuabsent in the base PyTorch build2d_regression/image_restoration.ipynb(transitively)transformers<5.0inrequirements-dev.txtmonai.networks.nets.restormernot present in this branch2d_regression/image_restoration.ipynbskip_run_papermillinrunner.shmonai.bundle.load()returnsOrderedDictin older local branch — local-only issue; upstream MONAI ≥1.5 already returnsnn.Moduleby defaultcomputer_assisted_intervention/endoscopic_inbody_classification.ipynbreturn_state_dict=Falsekwarg; original notebook is correctaimpackage not installed in the Docker imagemodules/spleen_segmentation_aim.ipynbaimtorequirements-dev.txtpytorch-lightning~=2.0.0(resolves to 2.0.9) eagerly importsmlflowat module level;mlflow 3.xfails on Python 3.12 with a relative-import errorbundle/05_spleen_segmentation_lightning.ipynbpytorch-lightning>=2.1in notebook (removes the eager mlflow import; tested with 2.6.5)mlflow<3.0as a safety net; this PR removes the fragile import dependency entirelymax_epochs; two notebooks have nonebundle/msd_crossval_datalist_generator.ipynb,pathology/hovernet_infer_compare.ipynbdoesnt_contain_max_epochsexemption list inrunner.shdeep_atlas_tutorial.ipynband05_spleen_segmentation_lightning.ipynbhardcodedevice = torch.device("cuda:0")/"cuda"with no CPU fallback; fail on the CPU-only GitHub Actions runnerdeep_atlas/deep_atlas_tutorial.ipynb,bundle/05_spleen_segmentation_lightning.ipynbskip_run_papermillinrunner.shpreprocess_detect_scene_and_split_fold.ipynb,deep_atlas_tutorial.ipynb,class_lung_lesion.ipynbautopep8autofix (no functional changes)Checks
./figurefolder./runner.sh -t <path to .ipynb file>— GPU-dependent notebooks (05_spleen_segmentation_lightning,deep_atlas_tutorial) require a CUDA device; they are skipped in the CPU CI runner and verified manually on a local GPU node.