Sinkhorn doubly stochastic attention rank decay analysis
Doubly stochastic attention slows rank collapse in Transformers — but skip connections still do the heavy lifting.
The Thesis
Transformer models process language and images by letting every token 'attend' to every other token via a mechanism called self-attention. A known failure mode is rank collapse: as you stack more layers, the representations of different tokens become nearly identical, destroying the model's ability to distinguish meaning. This paper asks whether replacing the standard normalization step (Softmax, which makes each row of the attention matrix sum to one) with Sinkhorn normalization (which makes both rows and columns sum to one, called doubly stochastic) delays that collapse. The theoretical answer is yes — rank decays more slowly — but the practical punchline is that skip connections (residual links that pass information around attention layers) remain the dominant fix. The result is a careful theoretical characterization, not a new architecture ready for deployment.
Catalyst
Rank collapse and entropy collapse in deep Transformers have become well-documented failure modes only in the last two to three years, with papers like 'Attention is Not Only a Weight' and related work on Softmax pathologies. Sinkhorn-based doubly stochastic attention was recently proposed as an empirical remedy, but a theoretical explanation for why it helps had not been derived. This paper fills that gap with a formal bound.
What's New
Prior work, particularly the 'Attention is Not All You Need' line of research, proved that pure Softmax self-attention matrices cause rank to collapse doubly exponentially with depth — meaning signal degrades catastrophically fast without residual connections. Separate empirical work proposed doubly stochastic attention (using the Sinkhorn algorithm, an iterative rescaling that enforces both row and column sums equal to one) and showed it improved performance, but did not explain the mechanism theoretically. This paper derives an explicit bound showing Sinkhorn attention also collapses doubly exponentially, but with a slower rate constant — and validates this on sentiment analysis and image classification benchmarks.
The Counter
The paper's central theoretical result — that Sinkhorn attention also collapses doubly exponentially — is actually a concession, not a victory. Both methods suffer the same qualitative failure; the difference is only in the rate constant, which may matter little in practice when residual connections are present. The empirical validation is limited to sentiment analysis and image classification, two relatively shallow tasks; it does not demonstrate that doubly stochastic attention improves large-scale language model training where Softmax is already heavily optimized. Sinkhorn normalization requires iterative rescaling at every attention computation, adding latency that the paper does not seriously benchmark against the gains. Practitioners already have cheaper fixes — residual connections, layer normalization, and attention temperature scaling — that achieve stability without the computational overhead. The theoretical bound itself applies only to 'pure' self-attention (no residuals, no feedforward layers), making its relevance to real architectures uncertain.
Longs
- GOOG/Alphabet — large internal Transformer research teams that could absorb architectural improvements
- ARM Holdings (ARM) — inference efficiency gains matter for edge chips where ARM architectures dominate
- BOTZ (Global Robotics & AI ETF) — broad exposure to AI hardware and software benefiting from more stable deep networks
Shorts
- Vendors selling proprietary attention-optimization kernels (e.g., Flash Attention-only tooling) — if doubly stochastic attention requires different memory access patterns, existing optimized kernels may not apply directly
Enablers (Picks & Shovels)
- PyTorch (open-source deep learning framework used for experiments in this and related work)
- Hugging Face Transformers library — standard toolkit for reproducing attention mechanism research
- arXiv preprint infrastructure — rapid dissemination that lets practitioners test theoretical bounds quickly
Private Watchlist
- Mistral AI — active open-model research that iterates on Transformer internals
- Together AI — infrastructure for fine-tuning and serving Transformer variants at scale
Resources
The Paper
The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal degradation across layers. In particular, it can induce rank collapse, resulting in increasingly uniform token representations, as well as entropy collapse, characterized by highly concentrated attention distributions. Recent work has highlighted the benefits of doubly stochastic attention as a form of entropy regularization, promoting a more balanced attention distribution and leading to improved empirical performance. In this paper, we study rank collapse across network depth and show that doubly stochastic attention matrices normalized with Sinkhorn algorithm preserve rank more effectively than standard Softmax row-stochastic ones. As previously shown for Softmax, skip connections are crucial to mitigate rank collapse. We empirically validate this phenomenon on both sentiment analysis and image classification tasks. Moreover, we derive a theoretical bound for the pure self-attention rank decay when using Sinkhorn normalization and find that rank decays to one doubly exponentially with depth, a phenomenon that has already been shown for Softmax.