Temporal Regularized Learning: Self-Supervised Learning Local In Space And Time

Paper PDF

Temporal Regularized Learning (TeReL) is a highly local and self-supervised procedure that optimizes each neuron individually. We adapt the self-supervised loss formulation of VICReg, consisting of variance, invariance and covariance to input streams with sequential coherence and for online- compatibility. It removes the need for biphasic updates, negatives or inner-loop convergence, given three scalar memory units per neuron and an auxiliary lateral network. Knowledge about downstream tasks can be injected through the sequence ordering, allowing for supervised training. We present TeReL and its simplified variant, TeReL-S. Experiments on MNIST show TeReL is competitive with backpropagation, Forward-Forward and Equilibrium Propagation, while TeReL-S achieves similar performance despite its simplified setup. We show that TeReL creates neurons with specialized receptive fields in the first layer. In later layers, some neurons specialize by activating only for some types of input.

Cite the paper:

@misc{Wiest2025,
  author       = {Wiest, Davide},
  title        = {{Temporal Regularized Learning: Self-supervised learning local in space and time}},
  publisher    = {Zenodo},
  year         = {2026},
  doi          = {10.5281/zenodo.18673107},
  url          = {https://doi.org/10.5281/zenodo.18673107}
}

Quickstart

• Train a TeReL model with train.py.
• terel/config/configurations.py contains functions that modify the setup, e.g. modify the model architecture.
• The previous_versions folder has a README with short explanations of what changed in each version.

The paper experiments were run on earlier commits than the current one. MNIST Classification

• Backprop: 25908634afa795840b0026d8481fa69338e857ec
• TeReL / TeReL-S: b39b73fac94e984f089820bec7a421499bcd6c0d

MNIST Rows next-row prediction:

• 0e454ac (rnn and new analysis outputs)
• 221f22c (rnn setup)

Current code has changed since these commits.