DeltaMask

Minimizing Communication in Federated Fine-Tuning
of Vision Foundation Models

Vasileios Tsouvalas · Yuki M. Asano · Aaqib Saeed

Abstract

DeltaMask adapts vision foundation models in Federated Learning (FL) while keeping pretrained weights frozen. Clients learn stochastic binary masks, compare them with a shared reference, and retain the changes with the highest information gain. Four-wise binary fuse filters encode those positions into compact PNG uploads. The server approximately reconstructs the masks and updates their probabilities through Bayesian aggregation. Across eight datasets and five pretrained backbones, the paper reports communication as low as 0.09 bits per parameter, with competitive recognition accuracy.

Setup

Create and activate the deltamask environment:

git clone https://github.com/Nousphera/DeltaMask.git
cd DeltaMask
python3.12 -m venv --prompt deltamask "$HOME/.venvs/deltamask"
source "$HOME/.venvs/deltamask/bin/activate"

From the activated deltamask environment, install the requirements:

python -m pip install -r requirements.txt

Note

The bundled four-wise filter builds with a C++17 compiler: Command Line Tools on macOS, or build-essential python3-dev on Ubuntu.

Data and Weights

Download, prepare and validate the datasets:

# One dataset
python scripts/prepare_data.py --dataset cifar10

# All datasets
python scripts/prepare_data.py

Datasets: cifar10 · cifar100 · svhn · fashion-mnist · emnist · food101 · eurosat · cars196.

For Cars196, download the images, devkit and test labels into raw/cars196/stanford_cars/ before preparation:

stanford_cars/
├── cars_train/
├── cars_test/
├── cars_test_annos_withlabels.mat
└── devkit/
    ├── cars_train_annos.mat
    └── cars_meta.mat

Note

EuroSAT uses an 80/20 split with seed 42. EMNIST defaults to letters; select --emnist-split balanced for the balanced variant.

Prepare Model

python scripts/prepare_model.py

Backbones: clip-vit-b32 (default) · clip-vit-l14 · dinov2-base · dinov2-small · convmixer-768-32. Select one with --model, or use --model all.

Run Experiments

From the repository root, launch your chosen method as a Flower App:

# Shared experiment settings
common=(--dataset cifar10 --model clip-vit-b32
        --nr-clients 30 --rounds 100 --participation 1.0
        --alpha 10 --batch-size 64 --device cuda)

# DeltaMask (ours)
python scripts/run_experiment.py "${common[@]}" --method deltamask

# Fine-tuning · FedAvg
python scripts/run_experiment.py "${common[@]}" --method finetune

# Linear probing · frozen backbone
python scripts/run_experiment.py "${common[@]}" --method linear-probe

# FedPM · stochastic masks, packed-bit transport
python scripts/run_experiment.py "${common[@]}" --method fedpm

# FedMask · threshold-training variant
python scripts/run_experiment.py "${common[@]}" --method fedmask

Note

All methods share a one-round classifier warm-up; the mask methods adapt the final five blocks, projection and classifier. FedPM uses lossless packed bits; FedMask follows the paper experiment's threshold-training variant with stochastic Bayesian aggregation.

Citation

@inproceedings{tsouvalas2025deltamask,
  title = {DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models},
  author = {Vasileios Tsouvalas and Yuki M. Asano and Aaqib Saeed},
  booktitle = {2024 IEEE International Conference on Big Data (BigData)},
  year = {2025},
  pages = {8042--8051},
  publisher = {IEEE},
  doi = {10.1109/BigData62323.2024.10825398},
  url = {https://doi.org/10.1109/BigData62323.2024.10825398}
}

License