Add the end-to-end runner, chest X-ray as a third modality, and run provenance

[Rian354]

Summary

The end-to-end runner, chest X-ray as a third modality, and run provenance.

With the corrections in #46 and the encoders in #48, this loop produces the primary result of the
project. Notes and laboratory values together give PR-AUC 0.4062, against 0.2956 for
laboratory values alone. The paired difference is +0.1106, with a 95% interval of
+/- 0.0030 over three seeds.

Both arms use an identical backbone: 128 embedding dimensions, 128 hidden units, 2 layers, 4 heads.
The comparison is matched on parameters and depth. Only the token count is different.

One limit is stated in full below: the note-presence confound is not yet controlled, so this
result shows that notes_labs is better than labs_only, and not yet that the note content is the
reason.

The results below predate the padding fix in #48. That fix changes what the model reads, so
every number in this description must be produced again once #48 merges. They are kept here because
they document the pipeline and the controls, not because they are final.

Depends on #46 and #48. The runner imports sample_oversample and LabsOnlyMIMIC4 from #46,
and fit_lab_standardizer from #48. Merge those first.

---

Implemented

1. Chest X-ray as a third modality

• Files: pyhealth/tasks/multimodal_mimic4.py, pyhealth/datasets/mimic4.py,
  pyhealth/datasets/configs/mimic4_cxr.yaml
• Added CXRMultimodalMIMIC4 with the arms cxr_only, cxr_labs and cxr_notes_labs.
• MIMIC-CXR has no hadm_id. The task links an image to an admission only when StudyDate and
  StudyTime of the same subject fall inside the observation interval of that admission.
• An image event therefore has a real position on the timeline, in hours from admission. This is
  the same convention that laboratory values use, so the unified embedding reads every modality the
  same way.
• MIMIC dates move into the future for each patient, but MIMIC-IV admission times move by the same
  amount for the same patient. The difference is therefore correct.
• prepare_metadata reuses a prepared metadata file when one is present. To rewrite a 100+ MB CSV
  for every CXR experiment is needless shared-filesystem IO, and it races a concurrent reader. An
  installation with raw metadata only still uses the builder.

2. Run provenance

• Files: pyhealth/utils.py, examples/mortality_prediction/unified_embedding_e2e_mimic4.py
• metrics_history.json records the score of a run but not the conditions. A run with a frozen
  encoder and a run with a trainable encoder look identical after the job output is gone. This is
  the reason that the frozen state of earlier results cannot be recovered.
• Added write_run_config(), which writes run_config.json beside the metrics.
• The file records resolved settings and not the raw flags. This is important:
  --freeze-encoder is an alias, so a run can record text_finetune_mode="full" and train with a
  frozen encoder. The raw flag gives the wrong description.
• Code identity is a git commit and a SHA-256 digest of the package source. A cluster run starts
  from an unpacked archive, where the git command gives no result.
• The write is atomic, and a value that JSON cannot hold does not lose the whole record. A run that
  finishes with no provenance is the case this feature exists to prevent.

3. Two fallbacks that changed the measurement in silence

• File: examples/mortality_prediction/unified_embedding_e2e_mimic4.py
• _split_dataset changed from split_by_patient to split_by_sample when the patient split was
  empty. That fallback leaks, because the admissions of one patient can then be in the train split
  and the test split. It occurs with a small cohort, which is the scale used for pipeline tests. It
  now gives a warning and records split_mode.
• The reported predictions came from test_loader or val_loader or train_loader. With no test
  split, the run reported validation performance, or training performance, as test performance. It
  now gives a warning and records eval_split.

4. Report of note availability

• File: examples/mortality_prediction/unified_embedding_e2e_mimic4.py
• A missing note is a constant placeholder embedding, so "the sample has a note" is easy to learn.
• Runs that record this report a ratio between 2.2x and 2.7x on the MIMIC-IV train split. The
  association is real and it is large.
• _note_availability_report() measures this on the train split only. It strides across the split
  rather than reading a prefix, because samples are grouped by patient and a prefix is not
  representative. It prints the result, records it in run_config.json, and warns above a ratio of
  1.5.

5. Discriminative learning rate for the text pathway

• File: pyhealth/trainer.py
• Added encoder_lr, which gives a pretrained text encoder a gentler rate than the randomly
  initialised layers around it.
• The subtle case is a frozen encoder. Then every embedding_model.encoders.* parameter has
  requires_grad=False, so a group matched on that prefix alone is empty, and encoder_lr
  controls nothing, while the only trainable text parameters, projections.*, keep the base rate.
• The projection therefore joins the group only when the encoder is frozen. Ordinary
  discriminative fine-tuning still gives the base rate to a projection with random values.

6. Loss trajectory for each epoch

• File: pyhealth/trainer.py
• Added train_loss_first_step, train_loss_first100 and train_loss_last100.
• A mean for one epoch cannot show the difference between a run that starts badly and a run that
  becomes worse inside the epoch. This difference cost significant diagnostic time.

7. The run directory name includes the task

• File: examples/mortality_prediction/unified_embedding_e2e_mimic4.py
• The name was f"{model}_seed{seed}". A paired comparison holds both of those fixed and varies
  the task, so --task labs_only and --task notes_labs at seed 42 both resolved to
  transformer_seed42. The second run overwrote the first run's metrics_history.json,
  run_config.json and predictions CSV.
• The loss is silent. The surviving directory looks like a complete run, and the provenance added
  in contribution 2 would describe only the arm that finished last.
• The name is now f"{task}_{model}_seed{seed}".

8. The runner matches this checkout's APIs

Caught by a smoke run on real MIMIC-IV. The unit tests did not build a task from parsed
arguments.

• NotesLabsMIMIC4 on this branch does not take include_labs, note_extraction,
  note_source, discharge_note_policy or text_normalize. The runner passed all of them, so
  --task notes_labs died at construction. It now passes only the parameters the class declares,
  and a flag the class cannot honour stops the run instead of being dropped.
• UnifiedMultimodalEmbeddingModel takes freeze_text_encoder, not text_finetune_mode.
  use_amp / amp_dtype are parameters of Trainer.train, not of Trainer.
• DataLoader worker options arrive with Use fused attention, make the mask fill fp16-safe, validate the AMP dtype, expose DataLoader worker options #47. Expanding them as **loader_kwargs hid them from
  an AST check. The runner now passes only what the installed get_dataloader signature
  declares.
• binary_metrics_fn has no f1_opt. Model selection uses pr_auc, so dropping it does not
  change which checkpoint is chosen.

9. CXR layout variant and image directory

• MIMIC4Dataset takes cxr_variant. The runner never passed it, so every CXR run used the
  default layout. The resized set is the sunlab layout; the default config expects
  studytime_normalized, which that set does not have: KeyError: 'studytime_normalized'.
• The sunlab variant then required a directory literally named images. The resized set lives
  under resized_images, so all three CXR arms failed on a complete 377,110-image cohort. Both
  names are now accepted.

---

Validation

Primary result

Full scale MIMIC-IV in-hospital mortality: 144,586 train samples and 18,074 test samples. The split
is pinned across the arms. The metrics use the intersection of the test patients (n = 18,041),
because the two tasks accept slightly different cohorts (846 positives against 842). An
almost-paired comparison is not a paired comparison.

Metric	labs_only	notes_labs	Paired difference	95% interval
PR-AUC	0.2956 +/- 0.0041	0.4062 +/- 0.0045	+0.1106	+/- 0.0030
ROC-AUC	0.8241 +/- 0.0118	0.9098 +/- 0.0006	+0.0857	+/- 0.0285
Brier	0.0677 +/- 0.0010	0.0389 +/- 0.0039	-0.0288	+/- 0.0074
Log-loss	0.2482 +/- 0.0038	0.1494 +/- 0.0202	-0.0988	+/- 0.0479

Three seeds, prevalence 0.0467. The interval uses the t quantile for 2 degrees of freedom, which is
4.303, and not a normal quantile. With three seeds the difference is large.

PR-AUC is separable: the difference is approximately 37 times its interval. ROC-AUC, Brier and
log-loss all move in the same direction, but their intervals are wide at three seeds, so PR-AUC is
the metric that carries the claim.

Both runs use balanced sampling (balanced_ratio=1.0) and leave pos_weight unset. This is
recorded in run_config.json for all six runs.

Both arms used the same explicit settings, --embedding-dim 128 --hidden-dim 128 --num-layers 2 --heads 4. This is a matched comparison at one setting and not a tuned comparison. Neither arm
received a hyperparameter search.

The presence confound is NOT yet controlled

A note is absent for a part of the cohort, and a missing note is a constant placeholder embedding.
The runner measures the association and reports it, but no run yet separates the two explanations
of the +0.1106.

Two controls are needed, and neither has been run: a baseline that receives the presence indicator
alone, and a comparison restricted to complete cases, where presence is constant and carries no
signal. Until those run, the correct statement is that notes_labs is better than labs_only, and
not that the note content is the reason.

Chest X-ray

Complete cohort: 377,110 of 377,110 images present, 0 dropped. The physionet directory on the
cluster contains approximately 23% of the images. p10 and p11 are complete, p12 is at 24.5%, and
p13 to p19 are absent, so a run that reached a p13 patient stopped on a missing file. These runs use
the complete resized set, which is 256x256 greyscale and 3.3 GB.

Same backbone. One cohort, so the arms are paired. 18,542 train samples and 2,285 test samples, prevalence 0.0565, split seed 42 pinned, 6 epochs.

Arm	PR-AUC	ROC-AUC
cxr_only	0.0602	0.5267
cxr + labs	0.3082	0.8047
cxr + notes + labs	0.4096	0.8625

Each additional modality improves the result. Three limits: one seed for each arm, a CXR cohort that
is 13% of the full cohort because it needs a study inside the window, and 6 epochs against 20 for
the primary table. These numbers compare with each other and not with the primary table.

cxr_only is close to the prevalence (0.0602 against 0.0565). The advantage comes from the
combination of modalities and not from the images alone.

Provenance, verified on the six runs of the primary table

split_seed=42 pinned=True   split_mode=by_patient   eval_split=test
n_train=144586  n_test=18074
source_sha256=19bab0f4b875aef7   (identical for all six runs)

The splits and the code digest are identical, so the comparison is paired. A difference cannot come
from the composition of the split or from a change to the code.

---

Unit tests

File: tests/test_run_directory_naming.py, 5 tests. Three fail on the pre-fix commit.

File: tests/test_run_provenance_and_pathways.py.

Test group	Purpose
Provenance (5)	The file is written beside the metrics, code identity survives a run from an archive, the digest is stable, a value that JSON cannot hold does not lose the record, and no temporary file remains
Text pathway (4)	A trainable encoder keeps the projection at the base rate, a frozen encoder puts it in the group, a non-text parameter never joins, and one field name does not capture another
Loss trajectory (1)	The three fields are recorded
Chest X-ray	Each arm declares the fields it uses, the lab mask matches the other tasks, an image uses the time convention of a laboratory value, and the sunlab layout accepts resized_images as well as images

These pass on this branch and fail on main.

No regression

The full suite was run on main, and on main with #46, #48 and this PR applied.

Tree	Passed	Failed
main	873	40
main + #46 + #48 + this PR	958	33

Every one of the 33 remaining failures is also present on main. They come from the environment:
test_tfm_tokenizer and test_tuple_time_text_tokenizer cannot build a fast tokenizer without
sentencepiece, and test_audio_processor cannot load torchaudio against this build of torch.

---

Not in this PR

Two fusion baselines, CrossAttentionFusion and SimpleLateFusion, are held for a later PR. They
are separate architectures, and no result in this PR uses them.

Merge notes

• This PR and Fix observation window leakage in lab collection, remove duplicate ICDLabsMIMIC4, restore missing splitter imports #46 both append a task class to the end of pyhealth/tasks/multimodal_mimic4.py.
  Git reports a conflict there. Both classes are kept, and there is no shared code between them.
• PR Add MedFuse model + MIMIC-IV EHR/CXR mortality task + runner #44 (jc/medfuse-multimodal) adds a different MIMIC-IV EHR/CXR task. Two CXR tasks must not
  merge independently. Please contact the author of Add MedFuse model + MIMIC-IV EHR/CXR mortality task + runner #44 before merge.

Question for the reviewer

The discharge-note policy uses admission-context section extraction, from Lee et al. (2023).
Retrieval covers the admission, and the extraction of admission-context sections is the temporal
control. The summary is written with knowledge of the outcome, and extraction reduces but does not
remove this condition. The alternative permits no information from after the prediction time, but it
removes approximately 90% of the notes, and it makes the presence of a note equivalent to a short
length of stay, which correlates with the outcome. Which default is correct for the paper?

[Rian354]

The sunlab CXR loader writes the metadata CSV under cache_dir when the PhysioNet root is read-only. JambaEHR and the e2e CLI default to 2 transformer + 2 mamba layers.