Skip to content

Fix observation window leakage in lab collection, remove duplicate ICDLabsMIMIC4, restore missing splitter imports - #46

Open
Rian354 wants to merge 3 commits into
mainfrom
fix/observation-window-leakage
Open

Fix observation window leakage in lab collection, remove duplicate ICDLabsMIMIC4, restore missing splitter imports#46
Rian354 wants to merge 3 commits into
mainfrom
fix/observation-window-leakage

Conversation

@Rian354

@Rian354 Rian354 commented Aug 10, 2026

Copy link
Copy Markdown

Summary

Four task bodies calculate an observation window from window_hours. Then they ignore it and
collect laboratory values through discharge. For a mortality label this reads the outcome,
because laboratory values from the hours before death are almost deterministic.

The effect is large. With the window honoured, labs_only PR-AUC decreases from 0.6204 to
0.2137
, and ROC-AUC decreases from 0.90 to 0.7136. The leaky number is not a baseline. It
is the answer.

clinical_notes_icd_labs is the default TABLE2_TASK in scripts/slurm/run_table2.sh and
scripts/condor/run_table2.sh. This defect therefore affects the default experiment.

This PR also repairs two unconditional failures on main. The runner cannot import, and one task
class has two definitions.


Implemented

1. One helper for the observation window

  • File: pyhealth/tasks/multimodal_mimic4.py
  • Added _admission_window_end(admission_time, admission_dischtime) to
    BaseMultimodalMIMIC4Task.
  • The helper returns admission_time + window_hours and limits the result to the discharge time.
    If window_hours is None, the helper returns the discharge time. This keeps the whole-stay
    mode.
  • Routed all four call sites through the helper. On main these are lines 746, 889, 1055 and
    1221.
  • Line 1055 is in ClinicalNotesICDLabsCXRMIMIC4. The CXR-linked task had the same defect.

2. Correct anchor for the observation window

  • File: pyhealth/tasks/multimodal_mimic4.py
  • _compute_effective_window anchors on the first admission of the patient. The previous code
    used that one absolute end time for every admission.
  • A second admission 400 days later received a window that ended before it started. That
    admission collected no events, and the task then added a placeholder row.
  • Two results followed. First, the protocol was not "24 hours from admission". It was "the first
    24 hours of the FIRST admission, to predict death at any admission up to the fatal one".
    Second, the sequence length was (laboratory values in the first 24 hours) + (number of
    admissions - 1). The tensor thus contained the future admission count of the patient.
  • _admission_window_end anchors on each admission.

3. Removal of the duplicate task class

  • File: pyhealth/tasks/multimodal_mimic4.py
  • main defines ICDLabsMIMIC4 two times, at lines 803 and 1136. The two bodies are different.
    One uses MISSING_CODE_TOKEN and skips an admission that has no discharge time. The other uses
    MISSING_TEXT_TOKEN and keeps that admission.
  • Python binds the second definition. The __call__ code object has co_firstlineno = 1202,
    which is in the second class. inspect.getsourcelines reports the first class, but that
    function searches the source text by name and is not correct for duplicates.
  • The first definition is therefore dead code, and the removal does not change behaviour.
  • File: pyhealth/tasks/__init__.py. ClinicalNotesICDLabsMIMIC4 was imported two times.

4. Cache invalidation

  • File: pyhealth/tasks/multimodal_mimic4.py
  • The task cache key is uuid5 over {**vars(task), input_schema, output_schema}. See
    pyhealth/datasets/base_dataset.py.
  • A change to the code alone does not move the key. Every existing cache continues to supply the
    old samples, and no message reports this.
  • Added emitted_data_version to BaseMultimodalMIMIC4Task.__init__. Increase this value when
    the emitted data changes.

5. Restoration of two splitter functions

  • Files: pyhealth/datasets/splitter.py, pyhealth/datasets/__init__.py
  • examples/mortality_prediction/unified_embedding_e2e_mimic4.py imports sample_oversample and
    sample_weighted. main does not define them. The default Table-2 runner cannot start.
ImportError: cannot import name 'sample_oversample' from 'pyhealth.datasets'
  • The change is additive. The three *_tuh splitter functions of main are not modified.

6. New task class: LabsOnlyMIMIC4


Validation

Observation window

Arm PR-AUC ROC-AUC Seeds
labs_only, collection through discharge, split pinned 0.6204 0.90 5
labs_only, collection through discharge, split follows seed 0.6131 0.9030 5
labs_only, window honoured 0.2137 0.7136 1

Full scale: 144,586 train samples and 18,074 test samples, 20 epochs. The honest arm uses the
pinned split with one seed. Compare it against the pinned leaky arm (0.6204).

A window sweep measured nothing

A sweep of --observation-window-hours over 24, 48 and 96 produced three identical datasets. The
three cache keys were different, but the emitted data was the same, because window_hours was
inert.

Noise floor

Five seeds at full scale, run two times. The first run lets the patient split follow the seed.
The second run pins the split.

Design sd(PR-AUC) Detectable effect at 5 seeds
Split follows the seed 0.0210 ~0.018
Split pinned (paired) 0.0093 ~0.008

Most of the variation comes from the composition of the test set. Across the five splits, test
prevalence was between 4.50% and 4.92%. The correlation between test prevalence and PR-AUC is
r = 0.87. The random baseline of PR-AUC is the prevalence.

A single-seed difference therefore has a 95% interval of approximately +/- 0.06. The
pretrained-versus-scratch differences reported before this PR were between -0.063 and +0.022.
Those differences were inside the noise of the measurement.

Residual nondeterminism is 0.0003. This value comes from the same seed under identical
conditions in two arms.


Unit tests

File: tests/test_observation_window.py, 14 tests.

Test Purpose
test_every_lab_task_honours_its_observation_window Parameterised over all four tasks that emit laboratory values
test_observation_window_is_anchored_per_admission A later admission gets its own window, and a short stay is limited by discharge
test_window_change_invalidates_the_cache The version marker moves the cache key
test_window_none_still_collects_through_discharge The whole-stay mode is preserved
test_icd_labs_task_is_defined_once The duplicate class does not return

The 14 tests pass on this branch. The same 14 tests fail on main. They test the defect and not
the implementation.


Backward compatibility

  • The splitter change is additive.
  • window_hours=None keeps the previous behaviour.
  • The removal of the duplicate class does not change behaviour, because Python binds the second
    definition.
  • emitted_data_version causes a cache rebuild. This is the purpose of the marker.

Status

The runner imports correctly after this change. The measurements above come from full-scale runs
on MIMIC-IV. This PR claims no modality comparison. See the end-to-end PR for that comparison.

Four task bodies computed a window from window_hours and then passed admission_dischtime, so labs were collected through discharge. Also re-anchors the window per admission, removes a shadowed duplicate ICDLabsMIMIC4, and restores two splitter functions the runner imports. Measurements are in the PR description.
Rian354 and others added 2 commits August 17, 2026 21:17
The processors stuffed "[MISSING_TEXT]" or a pad visit so the fast tokenizer would not crash on []. BERT then embedded a constant whose presence tracked mortality. Empty input is now (0, ...) events, and the tests that required the fake row now require its absence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Task bodies injected empty-string notes, a pad ICD visit, a black CXR path, or a zero lab row when a modality was absent. Those constants survived into BERT. Collectors now return empty lists, NotesLabsMIMIC4 uses the per-admission window helper, and emitted_data_version is 3 so stale caches cannot serve the old rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Rian354

Rian354 commented Aug 18, 2026

Copy link
Copy Markdown
Author

Empty notes, codes, and images are zero-length tensors now, not a fake [MISSING_TEXT] row, a pad visit, or a black frame. The notes+labs task actually uses the observation window helper, and the cache version is 3 so old caches cannot be reused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant