Why Geometric Continuity Emerges in Deep Neural Networks

Code for the NeurIPS 2026 paper Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking (Kyungwon Jeong, Won-Gi Paeng, Honggyo Suh).

Paper: arXiv:2605.04971

Principal right singular vectors v_1 of the 16 layer weights, projected on three principal components. Before training they are random. After training they trace a smooth trajectory only when the network has both residual connections and a symmetry-breaking activation (Res+ReLU); removing the activation (Res+None) or the residual connections (NoRes+ReLU) loses most of the continuity.

Overview

The leading singular vectors of adjacent layers' weight matrices tend to point in similar directions, a property we call geometric continuity. The paper asks why it emerges.

• Two conditions (toy MLPs). Residual connections make adjacent layers receive similar gradients, so the cumulative gradient is coherent across layers (continuity of the cumulative gradient: Res+ReLU .96, Res+None .85, versus at most .16 without residual connections). A nonlinearity in the residual branch keeps all layers in a shared coordinate frame. With a linear branch (Res+None) the layers drift apart along rotational symmetry directions and the alignment is lost: Res+None keeps only .26 weight continuity despite its coherent gradients, against .96 for Res+ReLU.
• Both nonlinearity and symmetry breaking are needed. Linear activations that break rotational symmetry (a fixed diagonal scaling or permutation) stay at or below the Res+None level (.16, .14), and an exactly rotation-equivariant nonlinearity (EqNL) reaches less than half of ReLU's continuity (.44 vs .96). In the architectures studied, continuity requires a non-equivariant nonlinearity, and the activation's equivariance error tracks continuity across activations (Spearman +0.89 over the 12 nonlinear residual configurations).
• Projection-specific continuity in transformers. Q, K, Gate and Up, which read from the residual stream, are continuous in their input space (v_1); O and Down, which write to it, in their output space (u_1); V, with no nonlinearity on its own coordinates, shows only low continuity in either space. In Llama-3.1-8B: Q .82, K .77, Gate .80, O .89, Down .70, V .13.

Contents

continuity/              the library
  svd.py                 leading singular vectors of per-layer weights
  alignment.py           adjacent-layer alignment (continuity), sigma^2-weighted alignment, s_l
  equivariance.py        equivariance error and nonlinearity of an activation
  models.py              residual MLP (Res/NoRes; ReLU, GELU, SiLU, Tanh, LeakyReLU, EqNL; optional LayerNorm)
  mnist.py               MNIST loader (downloads the original files on first use)
scripts/
  train_mlp.py           train the MLP and measure continuity            (Table 1)
  analyze_llm.py         projection-specific continuity of a pretrained LLM (Table 2, Appendix A)
  plot_trajectory.py     layer-wise trajectories and adjacent-layer alignment (as in Figure 1)

Install

pip install -r requirements.txt

Tested with Python 3.11 and 3.13, PyTorch 2.6, 2.13 and 2.14 (GPU and CPU), NumPy 2.4–2.5 and matplotlib 3.11. The scripts can be run from any directory; no installation of the package is needed.

Definitions

For layer weights W_1, ..., W_L with leading right singular vectors v_1^(l) (input space) and left singular vectors u_1^(l) (output space), geometric continuity is the mean over adjacent layers of |cos(v_1^(l), v_1^(l+1))| (or of u_1). The equivariance error of an activation sigma, evaluated on the trained network's own pre-activations z, is E_R ||sigma(Rz) - R sigma(z)||^2 / ||sigma(z)||^2 over random rotations R (6 rotations, averaged over layers). See Sections 3 and 4.3 of the paper.

Toy MLP (Table 1)

python scripts/train_mlp.py --act relu                  # Res+ReLU
python scripts/train_mlp.py --act none                  # Res+None: residual, linear activation
python scripts/train_mlp.py --act relu --no-residual    # NoRes+ReLU
python scripts/train_mlp.py --act eqnl0                 # Res+EqNL: exactly SO(d)-equivariant, nonlinear
python scripts/train_mlp.py --act none --layernorm      # Res+None+LN

Other activations: gelu, silu, tanh, leaky<alpha> (e.g. leaky0.1), eqnl<alpha>. Each run trains for 20 epochs (Adam, lr 1e-3, batch 128; about 1.5 minutes on one GPU), downloads MNIST to data/mnist/ on first use (or pass --data <dir> with the four original idx files), and writes results/<config>_s<seed>.json and .npz, e.g. results/Res+ReLU_s42.json. The paper averages seeds 42, 123 and 777 (--seed). Mean over the three seeds:

config	acc	eq-err	W v_1	W u_1	sigma^2-WA	G-bar v_1	Delta_GW	s_l
Res+ReLU	.976	193	.959	.937	.249	.961	+.003	.564
Res+Tanh	.970	.635	.846	.058	.283	.915	+.069	.151
Res+Leaky(0.1)	.970	8.76	.911	.929	.163	.956	+.045	.389
Res+EqNL(0)	.957	0	.435	.548	.311	.480	+.045	.354
Res+None	.909	0	.257	.260	.061	.851	+.594	.261
Res+None+LN	.924	0	.665	.650	.315	.733	+.068	.741
NoRes+ReLU	.727	4.36	.360	.647	.088	.106	−.255	.590

G-bar v_1 is the continuity of the cumulative gradient sum_t G_l^(t), and Delta_GW = G-bar v_1 − W v_1. On the GPU used for the paper this code reproduces the paper's per-seed values exactly (checked for Res+ReLU, Res+None, Res+EqNL(0) and NoRes+ReLU at seed 42); other hardware may differ in the last digits.

Pretrained models (Table 2, Appendix A)

python scripts/analyze_llm.py --model meta-llama/Llama-3.1-8B

Reads only the safetensors weights (CPU, about 10 minutes for Llama-3.1-8B) and reports the v_1 and u_1 continuity of Q, K, V, O, Gate, Up, Down and of the OV composite W_O W_V. For Llama-3.1-8B:

	Q	K	V	O	Gate	Up	Down	OV
v_1 (input space)	.817	.771	.133	.012	.800	.611	.005	.268
u_1 (output space)	.024	.025	.024	.892	.571	.007	.697	.797

The model is gated on Hugging Face: accept its license and log in (hf auth login) first, or pass a local checkpoint directory with --model <dir> --name Llama-3.1-8B.

Plots

python scripts/plot_trajectory.py results/Res+ReLU_s42.npz results/Res+None_s42.npz results/NoRes+ReLU_s42.npz
python scripts/plot_trajectory.py results/Llama-3.1-8B.npz --key Q_v1 O_u1 V_v1

Each panel shows the per-layer vectors on their first three principal components (top) and |cos| between adjacent layers (bottom).

License

MIT License, Copyright (c) 2026 Hyntel Inc. See LICENSE.

Citation

@inproceedings{jeong2026geometric,
  title     = {Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking},
  author    = {Jeong, Kyungwon and Paeng, Won-Gi and Suh, Honggyo},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}