MPSL

Fine-Tuning Multimodal Transformers on Edge:
A Parallel Split Learning Approach

Timo Fudala · Vasileios Tsouvalas · Nirvana Meratnia

Abstract

Multimodal Parallel Split Learning (MPSL) moves transformer fine-tuning from resource-constrained clients to a shared server. Clients retain lightweight modality tokenizers and send representations for parallel server processing. Local objectives drive a single, sample-weighted server update, with gradients returned to clients. MPSL supports early and late fusion across vision, language and audio, covering question answering, sentiment and emotion recognition, action recognition, and image–text retrieval. It substantially reduces client computation, with task-dependent accuracy trade-offs.

Setup

Create and activate the mpsl environment:

git clone https://github.com/Nousphera/MPSL.git
cd MPSL
python3.11 -m venv --prompt mpsl .venv
source .venv/bin/activate

From the activated mpsl environment, install the requirements:

python -m pip install -r requirements.txt

Data and Weights

Download, extract and validate a dataset, including the pretrained base encoder:

# One dataset
python scripts/prepare_data.py --dataset coco-qa

# All datasets
python scripts/prepare_data.py

Datasets: coco-qa · coco-retrieval · flickr30k · t4sa · ucf101 · kinetics-sounds · meld.

For datasets requiring access, download the following files before running the same command:

Dataset	Download and place in raw/
Flickr30K	Images at flickr30k-images/<image>.jpg, or the archive as flickr30k-images.tar.gz. Annotations download automatically.
T4SA	b-t4sa_imgs.tar and raw_tweets_text.csv. Alternatively, extract under b-t4sa/ with data/, b-t4sa_train.txt, b-t4sa_test.txt and the CSV.

Note

Kinetics-Sounds scans several hundred GB of official K400 archives, retaining only the paper's clips. Image/text and AST audio tokenizers load on first use.

Prepare Model

python scripts/prepare_metatransformer.py

Run Experiments

From the repository root, set the shared options and run your chosen method:

# Shared COCO-QA settings
common=(--dataset coco-qa --batch_size 500 --nr_of_epochs 15
        --nr_of_last_encoder_blocks_to_finetune 6 --num_workers 0)
clients=(--nr_of_clients 100 --dataset_split_type noniid --dirichlet_alpha 0.1)

# MPSL (ours)
python -m mpsl.main_mpsl "${common[@]}" "${clients[@]}" \
  --test_num_workers 0 --save_file_name cocoqa-mpsl

# Centralized
python -m mpsl.main_centralized "${common[@]}" \
  --save_file_name cocoqa-centralized

# FedAvg
python -m mpsl.main_fl "${common[@]}" "${clients[@]}" \
  --save_file_name cocoqa-fedavg

# FedCLIP
python -m mpsl.main_fl "${common[@]}" "${clients[@]}" \
  --use_adapter_approach True \
  --include_image_adapter True --include_text_adapter True \
  --save_file_name cocoqa-fedclip

Note

Metrics and weights are saved to checkpoints/<save_file_name>.{json,pth}.

Citation

@misc{fudala2025finetuningmultimodaltransformersedge,
  title={Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach},
  author={Timo Fudala and Vasileios Tsouvalas and Nirvana Meratnia},
  year={2025},
  eprint={2502.06355},
  archivePrefix={arXiv},
  primaryClass={cs.DC},
  url={https://arxiv.org/abs/2502.06355}
}

MIT License