Fix observation window leakage in lab collection, remove duplicate ICDLabsMIMIC4, restore missing splitter imports - #46
Open
Rian354 wants to merge 3 commits into
Open
Conversation
Four task bodies computed a window from window_hours and then passed admission_dischtime, so labs were collected through discharge. Also re-anchors the window per admission, removes a shadowed duplicate ICDLabsMIMIC4, and restores two splitter functions the runner imports. Measurements are in the PR description.
This was referenced Aug 11, 2026
Open
Draft
Rian354
force-pushed
the
fix/observation-window-leakage
branch
2 times, most recently
from
August 13, 2026 19:07
2925bc6 to
3bd8a54
Compare
The processors stuffed "[MISSING_TEXT]" or a pad visit so the fast tokenizer would not crash on []. BERT then embedded a constant whose presence tracked mortality. Empty input is now (0, ...) events, and the tests that required the fake row now require its absence. Co-authored-by: Cursor <cursoragent@cursor.com>
Task bodies injected empty-string notes, a pad ICD visit, a black CXR path, or a zero lab row when a modality was absent. Those constants survived into BERT. Collectors now return empty lists, NotesLabsMIMIC4 uses the per-admission window helper, and emitted_data_version is 3 so stale caches cannot serve the old rows. Co-authored-by: Cursor <cursoragent@cursor.com>
Author
|
Empty notes, codes, and images are zero-length tensors now, not a fake |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four task bodies calculate an observation window from
window_hours. Then they ignore it andcollect laboratory values through discharge. For a mortality label this reads the outcome,
because laboratory values from the hours before death are almost deterministic.
The effect is large. With the window honoured,
labs_onlyPR-AUC decreases from 0.6204 to0.2137, and ROC-AUC decreases from 0.90 to 0.7136. The leaky number is not a baseline. It
is the answer.
clinical_notes_icd_labsis the defaultTABLE2_TASKinscripts/slurm/run_table2.shandscripts/condor/run_table2.sh. This defect therefore affects the default experiment.This PR also repairs two unconditional failures on
main. The runner cannot import, and one taskclass has two definitions.
Implemented
1. One helper for the observation window
pyhealth/tasks/multimodal_mimic4.py_admission_window_end(admission_time, admission_dischtime)toBaseMultimodalMIMIC4Task.admission_time + window_hoursand limits the result to the discharge time.If
window_hoursisNone, the helper returns the discharge time. This keeps the whole-staymode.
mainthese are lines 746, 889, 1055 and1221.
ClinicalNotesICDLabsCXRMIMIC4. The CXR-linked task had the same defect.2. Correct anchor for the observation window
pyhealth/tasks/multimodal_mimic4.py_compute_effective_windowanchors on the first admission of the patient. The previous codeused that one absolute end time for every admission.
admission collected no events, and the task then added a placeholder row.
24 hours of the FIRST admission, to predict death at any admission up to the fatal one".
Second, the sequence length was (laboratory values in the first 24 hours) + (number of
admissions - 1). The tensor thus contained the future admission count of the patient.
_admission_window_endanchors on each admission.3. Removal of the duplicate task class
pyhealth/tasks/multimodal_mimic4.pymaindefinesICDLabsMIMIC4two times, at lines 803 and 1136. The two bodies are different.One uses
MISSING_CODE_TOKENand skips an admission that has no discharge time. The other usesMISSING_TEXT_TOKENand keeps that admission.__call__code object hasco_firstlineno = 1202,which is in the second class.
inspect.getsourcelinesreports the first class, but thatfunction searches the source text by name and is not correct for duplicates.
pyhealth/tasks/__init__.py.ClinicalNotesICDLabsMIMIC4was imported two times.4. Cache invalidation
pyhealth/tasks/multimodal_mimic4.pyuuid5over{**vars(task), input_schema, output_schema}. Seepyhealth/datasets/base_dataset.py.old samples, and no message reports this.
emitted_data_versiontoBaseMultimodalMIMIC4Task.__init__. Increase this value whenthe emitted data changes.
5. Restoration of two splitter functions
pyhealth/datasets/splitter.py,pyhealth/datasets/__init__.pyexamples/mortality_prediction/unified_embedding_e2e_mimic4.pyimportssample_oversampleandsample_weighted.maindoes not define them. The default Table-2 runner cannot start.*_tuhsplitter functions ofmainare not modified.6. New task class:
LabsOnlyMIMIC4pyhealth/tasks/multimodal_mimic4.pylabevents. It uses nonotes and no ICD codes.
Validation
Observation window
labs_only, collection through discharge, split pinnedlabs_only, collection through discharge, split follows seedlabs_only, window honouredFull scale: 144,586 train samples and 18,074 test samples, 20 epochs. The honest arm uses the
pinned split with one seed. Compare it against the pinned leaky arm (0.6204).
A window sweep measured nothing
A sweep of
--observation-window-hoursover 24, 48 and 96 produced three identical datasets. Thethree cache keys were different, but the emitted data was the same, because
window_hourswasinert.
Noise floor
Five seeds at full scale, run two times. The first run lets the patient split follow the seed.
The second run pins the split.
Most of the variation comes from the composition of the test set. Across the five splits, test
prevalence was between 4.50% and 4.92%. The correlation between test prevalence and PR-AUC is
r = 0.87. The random baseline of PR-AUC is the prevalence.
A single-seed difference therefore has a 95% interval of approximately +/- 0.06. The
pretrained-versus-scratch differences reported before this PR were between -0.063 and +0.022.
Those differences were inside the noise of the measurement.
Residual nondeterminism is 0.0003. This value comes from the same seed under identical
conditions in two arms.
Unit tests
File:
tests/test_observation_window.py, 14 tests.test_every_lab_task_honours_its_observation_windowtest_observation_window_is_anchored_per_admissiontest_window_change_invalidates_the_cachetest_window_none_still_collects_through_dischargetest_icd_labs_task_is_defined_onceThe 14 tests pass on this branch. The same 14 tests fail on
main. They test the defect and notthe implementation.
Backward compatibility
window_hours=Nonekeeps the previous behaviour.definition.
emitted_data_versioncauses a cache rebuild. This is the purpose of the marker.Status
The runner imports correctly after this change. The measurements above come from full-scale runs
on MIMIC-IV. This PR claims no modality comparison. See the end-to-end PR for that comparison.