---
title: "A mathematical perspective on Transformers"
source: "https://arxiv.org/html/2312.10794v5"
author:
published:
created: 2026-03-08
description:
tags:
  - "clippings"
---
HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

- failed: mathalpha.sty

Authors: achieve the best HTML results from your LaTeX submissions by following these [best practices](https://info.arxiv.org/help/submit_latex_best_practices.html).

arXiv:2312.10794v5 \[cs.LG\] 21 Aug 2025

Borjan Geshkovski Department of Mathematics, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA [borjan@mit.edu](https://arxiv.org/html/), Cyril Letrouit CNRS & Université Paris-Saclay  
Laboratoire de mathématiques d’Orsay, 307 rue Michel Magat, Bâtiment 307, 91400 Orsay, France [cyril.letrouit@universite-paris-saclay.fr](https://arxiv.org/html/), Yury Polyanskiy Department of EECS, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA [yp@mit.edu](https://arxiv.org/html/) and Philippe Rigollet Department of Mathematics, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA [rigollet@math.mit.edu](https://arxiv.org/html/)

###### Abstract.

Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as interacting particle systems, with a particular emphasis on long-time clustering behavior. Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.

###### Key words and phrases:

Transformers, self-attention, interacting particle systems, clustering, gradient flows

###### 1991 Mathematics Subject Classification:

Primary: 34D05, 34D06, 35Q83; Secondary: 52C17

### 1\. Outline

The introduction of *Transformers* in 2017 by Vaswani et al. \[[VSP <sup><span>+</span></sup> 17](https://arxiv.org/html/2312.10794v5#bib.bibx122)\] marked a significant milestone in the development of neural network architectures. Central to this contribution is *self-attention*, a novel mechanism which distinguishes Transformers from traditional architectures, and which plays a substantial role in their superior practical performance. In fact, this innovation has been a key catalyst for the progress of artificial intelligence in areas such as computer vision and natural language processing, notably with the emergence of large language models. As a result, understanding the mechanisms by which Transformers, and especially self-attention, process data is a crucial yet largely uncharted research area.

A common characteristic of deep neural networks (DNNs) is their compositional nature: data is processed sequentially, layer by layer, resulting in a discrete-time dynamical system (we refer the reader to the textbook \[[GBC16](https://arxiv.org/html/2312.10794v5#bib.bibx55)\] for a general introduction). This perspective has been successfully employed to model *residual neural networks* —see Section  [2.1](https://arxiv.org/html/2312.10794v5#S2.SS1 "2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers") for more details—as continuous-time dynamical systems called neural ordinary differential equations (neural ODEs) \[[CRBD18](https://arxiv.org/html/2312.10794v5#bib.bibx31), [E17](https://arxiv.org/html/2312.10794v5#bib.bibx47), [HR17](https://arxiv.org/html/2312.10794v5#bib.bibx66)\]. In this context, an input $x(0)\in\mathbb{R}^{d}$ , say an image, is evolving according to a given time-varying velocity field as $\dot{x}(t)=v_{t}(x(t))$ over some time interval $(0,T)$ . As such, a DNN can be seen as a flow map $x(0)\mapsto x(T)$ from $\mathbb{R}^{d}$ to $\mathbb{R}^{d}$ . Even within the restricted class of velocity fields $\{v_{t}\}_{t\geq 0}$ imposed by classical DNN architectures, such flow maps enjoy strong approximation properties as exemplified by a long line of work on these questions \[[LJ18](https://arxiv.org/html/2312.10794v5#bib.bibx82), [ZGUA20](https://arxiv.org/html/2312.10794v5#bib.bibx129), [LLS22](https://arxiv.org/html/2312.10794v5#bib.bibx85), [TG22](https://arxiv.org/html/2312.10794v5#bib.bibx113), [RBZ23](https://arxiv.org/html/2312.10794v5#bib.bibx100), [CLLS23](https://arxiv.org/html/2312.10794v5#bib.bibx26)\].

Following \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\] and \[[VBC20](https://arxiv.org/html/2312.10794v5#bib.bibx116)\], we observe that Transformers are in fact flow maps on $\mathcal{P}(\mathbb{R}^{d})$ , the space of probability measures over $\mathbb{R}^{d}$ . To realize this flow map from measures to measures, Transformers evolve a *mean-field interacting particle system*. More specifically, every particle (called a *token* in this context) follows the flow of a vector field which depends on the empirical measure of all particles. In turn, the *continuity equation* governs the evolution of the empirical measure of particles, whose long-time behavior is of crucial interest. In this regard, our main observation is that particles tend to cluster under these dynamics. This phenomenon is of particular relevance in learning tasks such as *next-token prediction*, wherein one seeks to map a given input sequence (i.e., a sentence) of $n$ tokens (i.e., words) onto a given next token. In this case, the output measure encodes the probability distribution of the next token, and its clustering indicates a small number of possible outcomes. Our results on a simplified but insightful toy model indicate that the limiting distribution is actually a point mass, leaving no room for diversity or randomness, which is at odds with practical observations. This apparent paradox is resolved by the existence of a long-time metastable state. As can be seen from Figures [3](https://arxiv.org/html/2312.10794v5#S6.F3 "Figure 3 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [5](https://arxiv.org/html/2312.10794v5#S6.F5 "Figure 5 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), the Transformer flow appears to possess two different time-scales: in a first phase, tokens quickly form a few clusters, while in a second (much slower) phase, through the process of pairwise merging of clusters, all tokens finally collapse to a single point. This appears to corroborate behavior observed empirically in trained Transformer models, which goes under the names *token uniformity*, *over-smoothing* \[[CZC <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx35), [RZZD23](https://arxiv.org/html/2312.10794v5#bib.bibx102), [GWDW23](https://arxiv.org/html/2312.10794v5#bib.bibx60), [WAWJ24](https://arxiv.org/html/2312.10794v5#bib.bibx124), [WAW <sup><span>+</span></sup> 24](https://arxiv.org/html/2312.10794v5#bib.bibx123), [DBK24](https://arxiv.org/html/2312.10794v5#bib.bibx37), [SWJS24](https://arxiv.org/html/2312.10794v5#bib.bibx108)\], or *rank-collapse* \[[DCL21](https://arxiv.org/html/2312.10794v5#bib.bibx39), [FZH <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx54), [NAB <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx93), [JDB23](https://arxiv.org/html/2312.10794v5#bib.bibx69), [ZMZ <sup><span>+</span></sup> 23](https://arxiv.org/html/2312.10794v5#bib.bibx131), [ZLL <sup><span>+</span></sup> 23](https://arxiv.org/html/2312.10794v5#bib.bibx130), [NLL <sup><span>+</span></sup> 24](https://arxiv.org/html/2312.10794v5#bib.bibx94), [BHK24](https://arxiv.org/html/2312.10794v5#bib.bibx15), [CNQG24](https://arxiv.org/html/2312.10794v5#bib.bibx29)\]; see also Figure [1](https://arxiv.org/html/2312.10794v5#S2.F1 "Figure 1 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers").

The goal of this manuscript is twofold. On the one hand, we aim to provide a general and accessible framework to study Transformers from a mathematical perspective. In particular, the structure of these interacting particle systems enables concrete connections to established mathematical topics, such as nonlinear transport equations, Wasserstein gradient flows, collective behavior models, and optimal configurations of points on spheres, among others. On the other hand, we describe several promising research directions with a particular focus on the long-time clustering phenomenon. The main results we present are new, and we also provide what we believe are interesting open problems throughout the paper.

The rest of the paper is arranged in three parts.

##### Part: Modeling

We define an idealized model of the Transformer architecture that captures two of the main characteristics of transformers: *self-attention* and *layer-normalization*. Following a perspective put forward in classical architectures such as ResNets \[[CRBD18](https://arxiv.org/html/2312.10794v5#bib.bibx31), [E17](https://arxiv.org/html/2312.10794v5#bib.bibx47), [HR17](https://arxiv.org/html/2312.10794v5#bib.bibx66)\], we view the successive layers of a neural network as time discretizations of a dynamical system of interacting particles. Layer-normalization effectively constrains particles to evolve on the unit sphere $\mathbb{S}^{d-1}$ , whereas self-attention is the particular nonlinear coupling of the particles done through the empirical measure (Section [2](https://arxiv.org/html/2312.10794v5#S2 "2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). In turn, the empirical measure evolves according to the continuity partial differential equation (Section  [3](https://arxiv.org/html/2312.10794v5#S3 "3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). Even after significant simplifications, this toy model retains macroscopic characteristics of trained Transformers, namely clustering. We also introduce a simpler surrogate model for self-attention which has the convenient property of being a Wasserstein gradient flow \[[AGS05](https://arxiv.org/html/2312.10794v5#bib.bibx6)\] for an energy functional that is well-studied in the context of optimal configurations of points on the sphere and sheds a complementary light of the source of clustering.

##### Part: Clustering

In this part we recall existing and establish new mathematical results that indicate clustering of tokens in the long-time limit. (See Figure [2](https://arxiv.org/html/2312.10794v5#S3.F2 "Figure 2 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") for a summary.) Our first results (Theorem [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") in Section [4](https://arxiv.org/html/2312.10794v5#S4 "4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and Theorem [5.1](https://arxiv.org/html/2312.10794v5#S5.Thmtheorem1 "Theorem 5.1. ‣ 5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") in Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) are in extreme regimes of a temperature parameter $\beta^{-1}$ that appears in the equation. We then move to the high-dimensional case in Section [6](https://arxiv.org/html/2312.10794v5#S6 "6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), where we begin by recalling Theorem [6.1](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem1 "Theorem 6.1 ([MTG17, CRMB24]). ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") —a result of \[[MTG17](https://arxiv.org/html/2312.10794v5#bib.bibx92)\], recently revisited in \[[CRMB24](https://arxiv.org/html/2312.10794v5#bib.bibx32)\] —which entails long-time clustering at any temperature when $d\geq 3$ . We provide an exponential rate of convergence in Theorem  [6.3](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem3 "Theorem 6.3. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") when $d\geq n$ —here $n$ denotes the number of particles—. We complement this result with an even more precise characterization of the rate of contraction of particles into a cluster. Namely, we describe the histogram of all inter-particle distances, and the time at which all particles are already nearly clustered (Theorem  [6.9](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem9 "Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")).

##### Part: Further questions

We propose potential avenues for future research, largely in the form of open questions substantiated by numerical observations. We first focus on the case $d=2$ (Section  [7](https://arxiv.org/html/2312.10794v5#S7 "7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) and elicit a link to Kuramoto oscillators. We briefly show in Section  [9.1](https://arxiv.org/html/2312.10794v5#S9.SS1 "9.1. The repulsive case ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") how a simple and natural modification of our model leads to non-trivial questions related to optimal configurations on the sphere. The remaining sections explore interacting particle systems that allow for parameter tuning of the Transformers architectures, a key feature of practical implementations.

## Part I Modeling

We begin this part by presenting the mathematical model for a Transformer in Section [2](https://arxiv.org/html/2312.10794v5#S2 "2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"). Throughout the paper we focus on a simplified version that includes the self-attention mechanism as well as layer normalization (Section   [2.2](https://arxiv.org/html/2312.10794v5#S2.SS2 "2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), but excludes additional feed-forward layers commonly used in practice (Section  [2.3](https://arxiv.org/html/2312.10794v5#S2.SS3 "2.3. Toward the complete Transformer ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). This nonetheless leads to a highly nonlinear mean-field interacting particle system. In turn, this system implements, via the continuity equation, a flow map from initial to terminal distributions of particles that we present in Section  [3](https://arxiv.org/html/2312.10794v5#S3 "3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers").

### 2\. Interacting particle system

Before writing down the Transformer model, we first provide a brief preliminary discussion to clarify our methodological choice of treating the discrete layer indices in the model as a continuous time variable in Section [2.1](https://arxiv.org/html/2312.10794v5#S2.SS1 "2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), echoing previous work on ResNets. The specifics of the toy Transformer model are presented in Section  [2.2](https://arxiv.org/html/2312.10794v5#S2.SS2 "2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), and a complete model is presented in Section  [2.3](https://arxiv.org/html/2312.10794v5#S2.SS3 "2.3. Toward the complete Transformer ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers").

#### 2.1. Residual neural networks

One of the standard paradigms in machine learning is that of supervised learning, where one aims to approximate an unknown function $f:\mathbb{R}^{d}\to\mathbb{R}^{m}$ , from data, $\mathscr{D}=\{x^{(i)},f(x^{(i)})\}_{i\in[N]}$ say. This is typically done by choosing one among an arsenal of possible parametric models, whose parameters are then fit to the data by means of minimizing some user-specified cost. With the advent of graphical processing units (GPUs) in the realm of computer vision \[[KSH12](https://arxiv.org/html/2312.10794v5#bib.bibx76)\], large neural networks have become computationally accessible, resulting in their popularity as one such parametric model.

Within the class of neural networks, *residual neural networks* (ResNets for short) have become a staple DNN architecture since their introduction in \[[HZRS16](https://arxiv.org/html/2312.10794v5#bib.bibx68)\]. In their most basic form, ResNets approximate a function $f$ at $x\in\mathbb{R}^{d}$ through a sequence of affine transformations, a component-wise nonlinearity, and skip connections. Put in formulae,

<table><tbody><tr><td rowspan="1"><span>(2.1)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mi>x</mi> <mo></mo><mrow><mo>(</mo><mrow><mi>k</mi> <mo>+</mo> <mn>1</mn></mrow><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mrow><mi>x</mi> <mo></mo><mrow><mo>(</mo><mi>k</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mrow><mi>w</mi> <mo></mo><mrow><mo>(</mo><mi>k</mi><mo>)</mo></mrow> <mo></mo><mi>σ</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><mi>a</mi> <mo></mo><mrow><mo>(</mo><mi>k</mi><mo>)</mo></mrow> <mo></mo><mi>x</mi> <mo></mo><mrow><mo>(</mo><mi>k</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mrow><mi>b</mi> <mo></mo><mrow><mo>(</mo><mi>k</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd> <mtd><mrow><mrow><mtext>for </mtext><mo></mo><mi>k</mi></mrow> <mo>∈</mo> <mrow><mo>{</mo> <mn>0</mn><mo>,</mo><mi>…</mi><mo>,</mo><mrow><mi>L</mi> <mo>−</mo> <mn>1</mn></mrow> <mo>}</mo></mrow></mrow></mtd></mtr> <mtr><mtd><mrow><mrow><mrow><mi>x</mi> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>)</mo></mrow></mrow> <mo>=</mo> <mi>x</mi></mrow><mo>.</mo></mrow></mtd></mtr></mtable></mrow><annotation>\begin{dcases}x(k+1)=x(k)+w(k)\sigma(a(k)x(k)+b(k))&amp;\text{ for }k\in\{0,\ldots,L-1\}\\ x(0)=x\,.\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

Here $\sigma$ is a Lipschitz function applied component-wise to the input vector, while $\theta(\cdot)=(w(\cdot),a(\cdot),b(\cdot))\in\mathbb{R}^{d\times d}\times\mathbb{R}^{d\times d}\times\mathbb{R}^{d}$ are trainable parameters. We say that ([2.1](https://arxiv.org/html/2312.10794v5#S2.E1 "Equation 2.1 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) has $L\geq 1$ hidden layers (or $L+1$ layers, or is of depth $L$ ). The output $x_{i}(L)$ serves as a *representation* of the input $x^{(i)}$ that is then fed into a last layer that corresponds to a classical learning task such as linear or logistic regression in order to predict the label $f(x^{(i)})$ . One can also devise generalizations of ([2.1](https://arxiv.org/html/2312.10794v5#S2.E1 "Equation 2.1 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), for instance in which matrix-vector multiplications are replaced by discrete convolutions in order to reflect other common architectures such as convolutional neural networks \[[GBC16](https://arxiv.org/html/2312.10794v5#bib.bibx55), Chapter 9\]. The key element that all these models share is that they all have *skip-connections*, namely, the previous step $x_{i}(k)$ appears explicitly in the iteration for the next one.

One upside of ([2.1](https://arxiv.org/html/2312.10794v5#S2.E1 "Equation 2.1 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), which is the one of interest to our narrative, is that the layer index $k$ can naturally be interpreted as a time variable, motivating the continuous-time analogue

<table><tbody><tr><td rowspan="1"><span>(2.2)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mover><mi>x</mi> <mo>˙</mo></mover> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mi>w</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mi>σ</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><mi>a</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mi>x</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mrow><mi>b</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mtd> <mtd><mrow><mrow><mtext>for </mtext><mo></mo><mi>t</mi></mrow> <mo>∈</mo> <mrow><mo>(</mo><mn>0</mn><mo>,</mo><mi>T</mi><mo>)</mo></mrow></mrow></mtd></mtr> <mtr><mtd><mrow><mrow><mrow><mi>x</mi> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>)</mo></mrow></mrow> <mo>=</mo> <mi>x</mi></mrow><mo>.</mo></mrow></mtd></mtr></mtable></mrow><annotation>\begin{dcases}\dot{x}(t)=w(t)\sigma(a(t)x(t)+b(t))&amp;\text{ for }t\in(0,T)\\ x(0)=x.\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

These are dubbed *neural ordinary differential equations* (neural ODEs). Since their introduction in \[[CRBD18](https://arxiv.org/html/2312.10794v5#bib.bibx31), [E17](https://arxiv.org/html/2312.10794v5#bib.bibx47), [HR17](https://arxiv.org/html/2312.10794v5#bib.bibx66)\], neural ODEs have emerged as a flexible mathematical framework to implement and study ResNets. We use this abstraction here for convenience, as dynamical systems are more naturally defined in continuous time. Although this approach is helpful for exposition, we emphasize that all the results presented can also be derived in discrete time using the same techniques.

#### 2.2. The interacting particle system

Unlike ResNets, which operate on a single input vector $x(0)\in\mathbb{R}^{d}$ at a time, Transformers operate on a sequence of vectors of length $n$ , namely, $(x_{i}(0))_{i\in[n]}\in(\mathbb{R}^{d})^{n}$ . This perspective is rooted in natural language processing, where each vector represents a word, and the entire sequence a sentence or a paragraph. In particular, it allows to process words together with their context. A sequence element $x_{i}(0)\in\mathbb{R}^{d}$ is called a *token*, and the entire sequence $(x_{i}(0))_{i\in[n]}$ a *prompt*. We use the words “token” and “particle” interchangeably.

All practical implementations of Transformers make use of *layer normalization* \[[BKH16](https://arxiv.org/html/2312.10794v5#bib.bibx16)\], most commonly in the form of *root mean square (RMS) normalization* \[[ZS19](https://arxiv.org/html/2312.10794v5#bib.bibx132)\]. (See lines 105–116 in \[[Mis24](https://arxiv.org/html/2312.10794v5#bib.bibx89)\] for instance.) RMS normalization takes the sequence of tokens output after each layer, divides each token by its Euclidean norm <sup>1</sup> <sup>1</sup> 1 The original form instead consisted in an entry-wise standardization of every token, namely subtracting the mean of all tokens, then dividing by the standard deviation. (plus a small parameter to avoid a possible division by zero), and multiplies the result by a trained diagonal matrix. This process aims to ensure that tokens do not diverge, thus avoiding rounding errors and overflow. The result is an evolution on a time-varying axis-aligned ellipsoid. To simplify the presentation and obtain insight and precise results, we assume that the trained diagonal matrix is equal to the idenitity, so that we work on the unit sphere $\mathbb{S}^{d-1}$ throughout. This simplification is justified empirically in the trained ALBERT XLarge v2 model described in Figure  [1](https://arxiv.org/html/2312.10794v5#S2.F1 "Figure 1 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), wherein this diagonal matrix is constant over all layers, with entries of mean value equal to $0.44$ and standard deviation equal to $0.008$ . Furthermore, current text embeddings provided by OpenAI, namely text-embedding-3-small and text-embedding-3-large, return norm-one embedding vectors. While we can only speculate as to the actual implementation of these models, this is an indication that layer normalization could be as simple as the one used in our toy model.

A Transformer is then a flow map on $(\mathbb{S}^{d-1})^{n}$ : the input sequence $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ is an initial condition which is evolved through the dynamics

<table><tbody><tr><td rowspan="1"><span>(2.3)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>V</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\frac{1}{Z_{\beta,i}(t)}\sum_{j=1}^{n}e^{\beta\langle Q(t)x_{i}(t),K(t)x_{j}(t)\rangle}V(t)x_{j}(t)\right)</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $i\in[n]$ and $t\geq 0$ . (We refer the reader to ([2.8](https://arxiv.org/html/2312.10794v5#S2.E8 "Equation 2.8 ‣ 2.3.2. Feed-forward layers ‣ 2.3. Toward the complete Transformer ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) and Section [2.3](https://arxiv.org/html/2312.10794v5#S2.SS3 "2.3. Toward the complete Transformer ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers") for the full model.) Here and henceforth

|  | $$ \bm{\mathsf{P}}^{\perp}_{x}y=y-\langle x,y\rangle x $$ |  |
| --- | --- | --- |

denotes the projection of $y\in\mathbb{R}^{d}$ onto $\mathrm{T}_{x}\mathbb{S}^{d-1}$ . The *partition function* $Z_{\beta,i}(t)>0$ reads

<table><tbody><tr><td rowspan="1"><span>(2.4)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><munderover><mo>∑</mo> <mrow><mi>k</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup></mrow></mrow><mo>.</mo></mrow><annotation>Z_{\beta,i}(t)=\sum_{k=1}^{n}e^{\beta\langle Q(t)x_{i}(t),K(t)x_{k}(t)\rangle}.</annotation></semantics></math></td><td></td></tr></tbody></table>

where $(Q(\cdot),K(\cdot),V(\cdot))$ (standing for Query, Key, and Value) are parameter matrices learned from data, and $\beta>0$ a fixed number intrinsic to the model <sup>2</sup> <sup>2</sup> 2 In practical implementations the inner products are multiplied by $d^{-\frac{1}{2}}$ , which along with the typical magnitude of $Q,K$ leads to the appearance of $\beta$ ., which, can be seen as an inverse temperature using terminology from statistical physics. Note that $Q(\cdot),K(\cdot)$ need not be square.

The interacting particle system ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))–([2.4](https://arxiv.org/html/2312.10794v5#S2.E4 "Equation 2.4 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), a simplified version of which was first written down in \[[LLH <sup><span>+</span></sup> 20](https://arxiv.org/html/2312.10794v5#bib.bibx84), [DGCC21](https://arxiv.org/html/2312.10794v5#bib.bibx41), [SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\], importantly contains the true novelty that Transformers carry with regard to other models: the *self-attention mechanism*

<table><tbody><tr><td rowspan="1"><span>(2.5)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mrow><msub><mi>A</mi> <mrow><mi>i</mi> <mo></mo><mi>j</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>:=</mo> <mfrac><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac></mrow><mo>,</mo><mrow><mrow><mo>(</mo><mi>i</mi><mo>,</mo><mi>j</mi><mo>)</mo></mrow> <mo>∈</mo> <msup><mrow><mo>[</mo><mi>n</mi><mo>]</mo></mrow> <mn>2</mn></msup></mrow></mrow><mo>,</mo></mrow><annotation>A_{ij}(t):=\frac{e^{\beta\langle Q(t)x_{i}(t),K(t)x_{j}(t)\rangle}}{Z_{\beta,i}(t)}\,,\hskip 28.45274pt(i,j)\in[n]^{2},</annotation></semantics></math></td><td></td></tr></tbody></table>

which is the nonlinear coupling mechanism in the interacting particle system. The $n\times n$ stochastic matrix $A(t)$ (rows are probability vectors) is called the *self-attention matrix*. The wording *attention* stems from the fact that $A_{ij}(t)$ captures the attention given by particle $i$ to particle $j$ relatively to all particles $\ell\in[n]$ . In particular, a particle pays attention to its neighbors where neighborhoods are dictated by the matrices $Q(t)$ and $K(t)$ in ([2.5](https://arxiv.org/html/2312.10794v5#S2.E5 "Equation 2.5 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

It has been observed numerically that the probability vectors $(A_{ij}(\cdot))_{j\in[n]}$ (for $i\in[n]$ ) in a trained self-attention matrix exhibit behavior related to the syntactic and semantic structure of sentences in natural language processing tasks (see \[[VSP <sup><span>+</span></sup> 17](https://arxiv.org/html/2312.10794v5#bib.bibx122), Figures 3-5\]). To illustrate our conclusions as pedagogically as possible, throughout the paper we focus on a simplified scenario wherein the parameter matrices $(Q,K,V)$ are constant, and even all equal to the identity unless stated otherwise, resulting in the dynamics

<table><tbody><tr><td rowspan="1"><span>(SA)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\frac{1}{Z_{\beta,i}(t)}\sum_{j=1}^{n}e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}x_{j}(t)\right)</annotation></semantics></math></td><td></td></tr></tbody></table>

for $i\in[n]$ and $t\geq 0$ and, as before

<table><tbody><tr><td rowspan="1"><span>(2.6)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><munderover><mo>∑</mo> <mrow><mi>k</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup></mrow></mrow><mo>.</mo></mrow><annotation>Z_{\beta,i}(t)=\sum_{k=1}^{n}e^{\beta\langle x_{i}(t),x_{k}(t)\rangle}.</annotation></semantics></math></td><td></td></tr></tbody></table>

The appearance of clusters in Transformers is actually corroborated by numerical experiments with trained models (see Figure [1](https://arxiv.org/html/2312.10794v5#S2.F1 "Figure 1 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). While we focus on a much simplified model, numerical evidence shows that the clustering phenomenon looks qualitatively the same in the cases $Q=K=V=\lambda I_{d}$ , $\lambda>0$ , and generic random $(Q,K,V)$ (see Figures [3](https://arxiv.org/html/2312.10794v5#S6.F3 "Figure 3 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and  [5](https://arxiv.org/html/2312.10794v5#S6.F5 "Figure 5 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") for instance). We refer the interested reader directly to Sections  [4](https://arxiv.org/html/2312.10794v5#S4 "4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), [6](https://arxiv.org/html/2312.10794v5#S6 "6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"); here, we continue the presentation on the modeling of different mechanisms appearing in the Transformer architecture.

![Refer to caption](https://arxiv.org/html/x1.png)

Figure 1. Histogram of { ⟨ x i ( t ), j ⟩ } ∈ \[ n \] 2 ≠ \\{\\langle x\_{i}(t),x\_{j}(t)\\rangle\\}\_{(i,j)\\in\[n\]^{2},i\\neq j} at different layers in the context of the trained ALBERT XLarge v2 model ( LCG + 20 and https://huggingface.co/albert-xlarge-v2 4 contains all the mechanisms described in this text, namely, is a system of the form ( 2.8 ) (or rather the discretization thereof) with 12 or 24 layers. The sequence length is of the order of 512 1024, and the tokens evolve in ℝ 4096 \\mathbb{R}^{4096}. The dynamics are therefore high-dimensional, lending weight to assumptions made later on (Section 6 )., which has constant parameter matrices. Here we randomly selected a single prompt, which in this context is a paragraph from a random Wikipedia entry, and then generate the histogram of the pairwise inner products. We see the progressive emergence of clusters all the way to the th (and last) hidden layer (top), as evidenced by the growing mass at 1. If the number of layers is increased, up to 48 say, the clustering is further enhanced (bottom).

###### Remark 2.1 (Collective behavior).

The dynamics ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) have a strong resemblance to the vast literature on nonlinear systems arising in the modeling of collective behavior. In addition to the connection to the classical Kuramoto model describing synchronization of oscillators \[[Kur75](https://arxiv.org/html/2312.10794v5#bib.bibx77), [ABV <sup><span>+</span></sup> 05](https://arxiv.org/html/2312.10794v5#bib.bibx2)\] (made evident in Section [7.2](https://arxiv.org/html/2312.10794v5#S7.SS2 "7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")), Transformers are perhaps most similar to the Krause model \[[Kra00](https://arxiv.org/html/2312.10794v5#bib.bibx75)\]

|  | $$ \dot{x}_{i}(t)=\sum_{j=1}^{n}a_{ij}(x_{j}(t)-x_{i}(t)),\hskip 28.45274pta_{ij}=\frac{\phi(\\|x_{i}-x_{j}\\|^{2})}{\sum_{k=1}^{n}\phi(\\|x_{i}-x_{k}\\|^{2})}. $$ |  |
| --- | --- | --- | --- | --- | --- | --- |

which is non-symmetric in general ( $a_{ij}\neq a_{ji}$ ), much like ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). When $\phi$ is compactly supported, it has been shown in \[[JM14](https://arxiv.org/html/2312.10794v5#bib.bibx73)\] that the particles $x_{i}(t)$ assemble in several clusters as $t\rightarrow+\infty$ . Other related models include those of Vicsek \[[VCBJ <sup><span>+</span></sup> 95](https://arxiv.org/html/2312.10794v5#bib.bibx117)\], Hegselmann-Krause \[[HK02](https://arxiv.org/html/2312.10794v5#bib.bibx63)\] and Cucker-Smale \[[CS07](https://arxiv.org/html/2312.10794v5#bib.bibx33)\]. All these models exhibit a clustering behavior under various assumptions (see \[[MT14](https://arxiv.org/html/2312.10794v5#bib.bibx91), [Tad23](https://arxiv.org/html/2312.10794v5#bib.bibx110)\] and the references therein). Yet, none of the opinion dynamics models discussed above contain parameters appearing within the nonlinearity as in ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), whilst set on the sphere.

###### Remark 2.2 (Permutation equivariance).

A function $f:(\mathbb{S}^{d-1})^{n}\rightarrow(\mathbb{S}^{d-1})^{n}$ is permutation equivariant if $f(\pi X)=\pi(f_{1}(X),\ldots,f_{n}(X))$ for any $X\in(\mathbb{R}^{d})^{n}$ and for any permutation $\pi\in\mathbf{S}_{n}$ of $n$ elements. Otherwise put, if we permute the input $X$ , then the output $f(X)$ is permuted in the same way. Given $t>0$ , the Transformer ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), mapping $(x_{i}(0))_{i\in[n]}\mapsto(x_{i}(t))_{i\in[n]}$ , is permutation-equivariant on $(\mathbb{S}^{d-1})^{n}$ .

#### 2.3. Toward the complete Transformer

There are a couple of additional mechanisms used in practical implementations that we do not explicitly address or use in this study. The mathematical analysis of these mechanisms remains open.

##### 2.3.1. Multi-headed attention

Practical implementations spread out the computation of the self-attention mechanism at every $t$ through a sequence of *heads*, leading to the so-called *multi-headed self attention*. This consists in considering the following modification to ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")):

<table><tbody><tr><td rowspan="1"><span>(2.7)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><munderover><mo>∑</mo> <mrow><mi>h</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>H</mi></munderover> <mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><mfrac><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>Q</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>K</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi><mo>,</mo><mi>h</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><msub><mi>V</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\sum_{h=1}^{H}\sum_{j=1}^{n}\frac{e^{\beta\langle Q_{h}(t)x_{i}(t),K_{h}(t)x_{j}(t)\rangle}}{Z_{\beta,i,h}(t)}V_{h}(t)x_{j}(t)\right)</annotation></semantics></math></td><td></td></tr></tbody></table>

where $Z_{\beta,i,h}(t)$ is defined as in ([2.4](https://arxiv.org/html/2312.10794v5#S2.E4 "Equation 2.4 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) for the matrices $Q_{h}(t)$ and $K_{h}(t)$ . The integer $H\geq 1$ is called the number of heads <sup>5</sup> <sup>5</sup> 5 In practical implementations, $H$ is a divisor of $d$ , and the query and key matrices $Q_{h}(t)$ and $K_{h}(t)$ are $\frac{d}{H}\times d$ rectangular. This allows for further parallelization of computations and increased expressiveness. For mathematical purposes, we focus on working with arbitrary integers $H$ , and square weight matrices $Q_{h}$ and $K_{h}$ ..

The introduction of multiple heads also allows for drawing some interesting parallels with the literature on feed-forward neural networks, such as ResNets ([2.1](https://arxiv.org/html/2312.10794v5#S2.E1 "Equation 2.1 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). Considerable effort has been expended to understand $2$ -layer neural networks with width tending to $+\infty$ ; more precisely, consider ([2.1](https://arxiv.org/html/2312.10794v5#S2.E1 "Equation 2.1 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with $L=1$ , $w\in\mathbb{R}^{d\times\ell}$ , $a\in\mathbb{R}^{\ell\times d}$ , and $\ell\to+\infty$ . The infinite-width limit for Transformers is in fact very natural, as it is realized by stacking an arbitrary large number of heads: $H\to+\infty$ . Hence, the same questions as for $1$ -hidden layer neural networks may be asked: for instance the question of universal approximation, in the vein of \[[Cyb89](https://arxiv.org/html/2312.10794v5#bib.bibx34), [Bar93](https://arxiv.org/html/2312.10794v5#bib.bibx9)\].

##### 2.3.2. Feed-forward layers

The complete Transformer dynamics combines all of the above mechanisms with a feed-forward layer; in the discrete-time context, this is actually done by using a Lie-Trotter splitting scheme for

<table><tbody><tr><td rowspan="1"><span>(2.8)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mrow><munderover><mo>∑</mo> <mrow><mi>h</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>H</mi></munderover> <mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><mfrac><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>Q</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>K</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi><mo>,</mo><mi>h</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><msub><mi>V</mi> <mi>h</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow> <mo>+</mo> <mrow><mi>w</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mi>σ</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><mi>a</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mrow><mi>b</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\sum_{h=1}^{H}\sum_{j=1}^{n}\frac{e^{\beta\langle Q_{h}(t)x_{i}(t),K_{h}(t)x_{j}(t)\rangle}}{Z_{\beta,i,h}(t)}V_{h}(t)x_{j}(t)+w(t)\sigma(a(t)x_{i}(t)+b(t))\right),</annotation></semantics></math></td><td></td></tr></tbody></table>

where $w(t),a(t),b(t)$ and $\sigma$ are all as in ([2.2](https://arxiv.org/html/2312.10794v5#S2.E2 "Equation 2.2 ‣ 2.1. Residual neural networks ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). The interested reader is referred to \[[LLH <sup><span>+</span></sup> 20](https://arxiv.org/html/2312.10794v5#bib.bibx84), [PH22](https://arxiv.org/html/2312.10794v5#bib.bibx98)\] for all the details <sup>6</sup> <sup>6</sup> 6 and lines 123–130 in \[[Ope24](https://arxiv.org/html/2312.10794v5#bib.bibx95)\] for some relevant source code.. The feed-forward layers (convolutional layers can alternatively be considered) are of critical importance in applications and drive the existing results on approximation properties of Transformers \[[YBR <sup><span>+</span></sup> 19](https://arxiv.org/html/2312.10794v5#bib.bibx127)\]. Nevertheless, the analysis of this model is beyond the scope of our current methods.

### 3\. Measure to measure flow map

An important aspect of Transformers is that they are not hard-wired to take into account the order of the input sequence, contrary to other architectures used for natural language processing such as recurrent neural networks. In these applications, each token $x_{i}(0)\in\mathbb{R}^{d}$ contains not only a word embedding $w_{i}\in\mathbb{R}^{d}$ , but also an additional *positional encoding* (we postpone a discussion to Remark  [3.2](https://arxiv.org/html/2312.10794v5#S3.Thmtheorem2 "Remark 3.2 (Positional encoding). ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) which allows tokens to also carry their position in the input sequence. Therefore, an input sequence is perfectly encoded as a *set* of tokens $\{x_{1}(0),\ldots,x_{n}(0)\}$ , or equivalently as the empirical measure of its constituent tokens $\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}(0)}$ . Recall that the output of a Transformer is also a probability measure, namely $\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}(t)}$ , albeit one that captures the likelihood of the next token. As a result, one can view Transformers as flow maps between probability measures <sup>7</sup> <sup>7</sup> 7 See \[[DBPC19](https://arxiv.org/html/2312.10794v5#bib.bibx38), [VBC20](https://arxiv.org/html/2312.10794v5#bib.bibx116), [ZB21](https://arxiv.org/html/2312.10794v5#bib.bibx128)\] for further related work on neural networks acting on probability measures. on $\mathbb{S}^{d-1}$ . To describe this flow map, we appeal to the continuity equation, which governs precisely the evolution of the empirical measure of particles subject to dynamics. This perspective is already present in \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\], the only modification here being that we add the projection on the sphere arising from layer normalization.

After introducing the continuity equation in Section [3.1](https://arxiv.org/html/2312.10794v5#S3.SS1 "3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), we show that a particular interaction energy functional, which is maximized at any point mass, increases along solutions thereof in Section [3.2](https://arxiv.org/html/2312.10794v5#S3.SS2 "3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"). Motivated by this monotonicity property, in Section [3.3](https://arxiv.org/html/2312.10794v5#S3.SS3 "3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers") we propose an illustrative modified model which has the nice property of being a Wasserstein gradient flow for this energy. Finally, in Section [3.4](https://arxiv.org/html/2312.10794v5#S3.SS4 "3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), we demonstrate that the original equation presented in Section [3.1](https://arxiv.org/html/2312.10794v5#S3.SS1 "3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers") is itself a gradient flow for the same energy, upon changing the metric underlying the definition of the gradient.

#### 3.1. The continuity equation

The vector field driving the evolution of a single particle in ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) clearly depends on all $n$ particles. In fact, one can equivalently rewrite the dynamics as

<table><tbody><tr><td rowspan="1"><span>(3.1)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\dot{x}_{i}(t)=\mathcal{X}[\mu(t)](x_{i}(t))</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $i\in[n]$ and $t\geq 0$ , where

|  | $$ \mu(t,\cdot)=\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}(t)}(\cdot) $$ |  |
| --- | --- | --- |

is the empirical measure, while the vector field $\mathcal{X}[\mu]:\mathbb{S}^{d-1}\to\mathrm{T}\mathbb{S}^{d-1}$ reads

<table><tbody><tr><td rowspan="1"><span>(3.2)</span></td><td></td><td><math><semantics><mrow><mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mi>x</mi> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>μ</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><mo>∫</mo> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><mi>y</mi> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>y</mi> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>y</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\mathcal{X}[\mu](x)=\bm{\mathsf{P}}^{\perp}_{x}\left(\frac{1}{Z_{\beta,\mu}(x)}\int e^{\beta\langle x,y\rangle}y\,\mathrm{d}\mu(y)\right)</annotation></semantics></math></td><td></td></tr></tbody></table>

with

<table><tbody><tr><td rowspan="1"><span>(3.3)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>μ</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mo>∫</mo> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><mi>y</mi> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>y</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>.</mo></mrow><annotation>Z_{\beta,\mu}(x)=\int e^{\beta\langle x,y\rangle}\,\mathrm{d}\mu(y).</annotation></semantics></math></td><td></td></tr></tbody></table>

In other words, ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is a *mean-field interacting particle system*. The evolution of $\mu(t)$ is governed by the continuity equation <sup>8</sup> <sup>8</sup> 8 Unless stated otherwise, $\nabla$ and $\mathrm{div}$ henceforth stand for the spherical gradient and divergence respectively, and all integrals are taken over $\mathbb{S}^{d-1}$ .

<table><tbody><tr><td rowspan="1"><span>(3.4)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mi>μ</mi></mrow> <mo>+</mo> <mrow><mi>div</mi> <mo></mo><mrow><mo>(</mo><mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow> <mo></mo><mi>μ</mi></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow></mtd> <mtd><mrow><mrow><mtext>on </mtext><mo></mo><msub><mi>ℝ</mi> <mrow><mo>≥</mo> <mn>0</mn></mrow></msub></mrow> <mo>×</mo> <msup><mi>𝕊</mi> <mrow><mi>d</mi> <mo>−</mo> <mn>1</mn></mrow></msup></mrow></mtd></mtr> <mtr><mtd><mrow><msub><mi>μ</mi> <mrow><mo>|</mo> <mi>t</mi> <mo>=</mo> <mn>0</mn></mrow></msub> <mo>=</mo> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>)</mo></mrow></mrow></mrow></mtd> <mtd><mrow><mtext>on </mtext><mo></mo><msup><mi>𝕊</mi> <mrow><mi>d</mi> <mo>−</mo> <mn>1</mn></mrow></msup></mrow></mtd></mtr></mtable></mrow> <annotation>\begin{cases}\partial_{t}\mu+\mathrm{div}(\mathcal{X}[\mu]\mu)=0&amp;\text{ on }\mathbb{R}_{\geq 0}\times\mathbb{S}^{d-1}\\ \mu_{|t=0}=\mu(0)&amp;\text{ on }\mathbb{S}^{d-1}\end{cases}</annotation></semantics></math></td><td></td></tr></tbody></table>

satisfied in the sense of distributions.

###### Remark 3.1 (Well-posedness).

Global existence of weak, measure-valued solutions to ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) for arbitrary initial conditions $\mu(0)\in\mathcal{P}(\mathbb{S}^{d-1})$ follows by arguing exactly as in \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57), Lemma A.3\]. Here and henceforth, $\mathcal{P}(\mathbb{S}^{d-1})$ stands for the set of Borel probability measures on $\mathbb{S}^{d-1}$ .

###### Remark 3.2 (Positional encoding).

For the sake of completeness, in this brief segue we discuss a few ways to perform positional encoding. The original one, proposed in \[[VSP <sup><span>+</span></sup> 17](https://arxiv.org/html/2312.10794v5#bib.bibx122)\], proceeds as follows. Consider a sequence $(w_{i})_{i\in[n]}\in(\mathbb{R}^{d})^{n}$ of word embeddings. Then the positional encoding $p_{i}\in\mathbb{R}^{d}$ of the $i$ -th word embedding is defined as $(p_{i})_{2k}=\sin(\frac{i}{M^{2k/d}})$ and $(p_{i})_{2k+1}=\cos(\frac{i}{M^{2k/d}})$ for $k\in[d/2-1]$ , and $M>0$ is a user-defined scalar equal to $10^{4}$ in \[[VSP <sup><span>+</span></sup> 17](https://arxiv.org/html/2312.10794v5#bib.bibx122)\]. The $i$ -th token is then defined as the addition: $x_{i}(0)=w_{i}+p_{i}$ . Subsequent works simply use either a random <sup>9</sup> <sup>9</sup> 9 This rationale supports the assumption that initial tokens are drawn at random, which we make use of later on. positional encoding (i.e., $p_{i}$ is just some random vector) or a trained transformation. The addition can also be replaced with a concatenation $x_{i}(0)=[w_{i};p_{i}]$ . (See \[[LWLQ22](https://arxiv.org/html/2312.10794v5#bib.bibx87), [XZ23](https://arxiv.org/html/2312.10794v5#bib.bibx126)\] for details.)

###### Remark 3.3 (Mean field limit).

Although the analysis in this paper is focused on the flow of the empirical measure, one can also consider ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) for arbitrary initial probability measures $\mu(0)\in\mathcal{P}(\mathbb{S}^{d-1})$ . Both views can be linked through a mean-field limit-type result, which can be shown by making use of the Lipschitz nature of the vector field $\mathcal{X}[\mu]$ . The argument is classical and dates back at least to the work of Dobrushin \[[Dob79](https://arxiv.org/html/2312.10794v5#bib.bibx44)\]. Consider an initial empirical measure $\mu_{n}(0)=\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}(0)}$ , and suppose that the points $x_{i}(0)$ are such that $\lim_{n\to+\infty}W_{1}(\mu_{n}(0),\mu(0))=0$ for some probability measure $\mu(0)\in\mathcal{P}(\mathbb{S}^{d-1})$ . (Here $W_{1}$ denotes 1-Wasserstein distance—see \[[Vil09](https://arxiv.org/html/2312.10794v5#bib.bibx120)\] for definitions—.) Consider the solutions $\mu_{n}(t)$ and $\mu(t)$ to ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with initial data $\mu_{n}(0)$ and $\mu(0)$ respectively. Dobrushin’s argument is then centered around the estimate

|  | $$ W_{1}(\mu_{n}(t),\mu(t))\leq e^{O(1)\|t\|}W_{1}(\mu_{n}(0),\mu(0)) $$ |  |
| --- | --- | --- | --- | --- |

for any $t\in\mathbb{R}$ , which in the case of ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) can be shown without much difficulty (see \[[Vil01](https://arxiv.org/html/2312.10794v5#bib.bibx119), Chapter 4, Section 1\] or \[[Gol16](https://arxiv.org/html/2312.10794v5#bib.bibx58), Section 1.4.2\]). This elementary mean-field limit result has a couple of caveats. First, the time-dependence is exponential. Second, if one assumes that the points $x_{i}(0)$ are sampled i.i.d. according to $\mu_{0}$ , then $W_{1}(\mu_{n}(0),\mu(0))$ converges to zero at rate $n^{-\frac{1}{d-1}}$ \[[Dud69](https://arxiv.org/html/2312.10794v5#bib.bibx45), [BLG14](https://arxiv.org/html/2312.10794v5#bib.bibx17)\], which deteriorates quickly when $d$ grows. Dimension-free convergence has been established in some cases, for instance by replacing the Wasserstein distance with a more careful choice of metric as in \[[HHL23](https://arxiv.org/html/2312.10794v5#bib.bibx62), [Lac23](https://arxiv.org/html/2312.10794v5#bib.bibx78)\] or more generally in \[[SFG <sup><span>+</span></sup> 12](https://arxiv.org/html/2312.10794v5#bib.bibx105)\]. Similarly, the exponential time-dependence might also be improved, as recent works in the context of flows governed by Riesz/Coulomb singular kernels, with diffusion, can attest \[[RS23](https://arxiv.org/html/2312.10794v5#bib.bibx101), [GLBM24](https://arxiv.org/html/2312.10794v5#bib.bibx56)\] (see \[[LLF23](https://arxiv.org/html/2312.10794v5#bib.bibx83)\] for a result in the smooth kernel case). We do not address this question in further detail here. For more references on this well-established topic, the reader is referred to \[[Vil01](https://arxiv.org/html/2312.10794v5#bib.bibx119), [Gol16](https://arxiv.org/html/2312.10794v5#bib.bibx58), [Ser20](https://arxiv.org/html/2312.10794v5#bib.bibx104)\] and the references therein.

#### 3.2. The interaction energy

One can naturally ask whether the evolution in ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) admits some quantities which are monotonic when evaluated along the flow. As it turns out, the *interaction* *energy*

<table><tbody><tr><td rowspan="1"><span>(3.5)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mi>𝖤</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><mn>2</mn> <mo></mo><mi>β</mi></mrow></mfrac> <mo></mo><mrow><mo>∬</mo> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><msup><mi>x</mi> <mo>′</mo></msup><mo>)</mo></mrow></mrow></mrow></mrow></mrow> <annotation>\mathsf{E}_{\beta}[\mu]=\frac{1}{2\beta}\iint e^{\beta\langle x,x^{\prime}\rangle}\,\mathrm{d}\mu(x)\,\mathrm{d}\mu(x^{\prime})</annotation></semantics></math></td><td></td></tr></tbody></table>

is one such quantity. Indeed,

<table><tbody><tr><td></td><td><math><semantics><mrow><mstyle><mfrac><mi>d</mi> <mrow><mi>d</mi> <mo></mo><mi>t</mi></mrow></mfrac></mstyle> <mo></mo><msub><mi>𝖤</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow></mrow> <annotation>\displaystyle\frac{\,\mathrm{d}}{\,\mathrm{d}t}\mathsf{E}_{\beta}[\mu(t)]</annotation></semantics></math></td><td><math><semantics><mrow><mo>=</mo> <mstyle><mrow><mo>∬</mo> <mrow><msup><mi>β</mi> <mrow><mo>−</mo> <mn>1</mn></mrow></msup> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>d</mi> <mo></mo><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow> <mo></mo><mi>d</mi> <mo></mo><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mstyle></mrow> <annotation>\displaystyle=\iint\beta^{-1}e^{\beta\langle x,x^{\prime}\rangle}\,\mathrm{d}\partial_{t}\mu(t,x)\,\mathrm{d}\mu(t,x^{\prime})</annotation></semantics></math></td><td></td></tr></tbody><tbody><tr><td></td><td></td><td><math><semantics><mrow><mo>=</mo> <mstyle><mrow><mo>∫</mo> <mrow><mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>⋅</mo> <mrow><mo>∫</mo> <mrow><mrow><mrow><mo>∇</mo> <mrow><mo>(</mo><mrow><msup><mi>β</mi> <mrow><mo>−</mo> <mn>1</mn></mrow></msup> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup> <mo>⟩</mo></mrow></mrow></msup></mrow><mo>)</mo></mrow></mrow> <mo>⁡</mo> <mrow><mi>d</mi> <mo></mo><mi>μ</mi></mrow></mrow> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup><mo>)</mo></mrow> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mstyle></mrow> <annotation>\displaystyle=\int\mathcal{X}[\mu(t)](x)\cdot\int\nabla\left(\beta^{-1}e^{\beta\langle x,x^{\prime}\rangle}\right)\,\mathrm{d}\mu(t,x^{\prime})\,\mathrm{d}\mu(t,x)</annotation></semantics></math></td><td></td></tr></tbody><tbody><tr><td rowspan="1"><span>(3.6)</span></td><td></td><td></td><td><math><semantics><mrow><mo>=</mo> <mstyle><mrow><mo>∫</mo> <mrow><msup><mrow><mo>‖</mo> <mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>‖</mo></mrow> <mn>2</mn></msup> <mo></mo><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow></mstyle></mrow> <annotation>\displaystyle=\int\Big{\|}\mathcal{X}[\mu(t)](x)\Big{\|}^{2}Z_{\beta,\mu(t)}(x)\,\mathrm{d}\mu(t,x)</annotation></semantics></math></td><td></td></tr></tbody></table>

for any $t\geq 0$ by using integration by parts. Recalling the definition of $Z_{\beta,\mu}(x)$ in ([3.3](https://arxiv.org/html/2312.10794v5#S3.E3 "Equation 3.3 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), we see that $e^{-\beta}\leq Z_{\beta,\mu}(x)\leq e^{\beta}$ for all $x\in\mathbb{S}^{d-1}$ . The identity ([3.2](https://arxiv.org/html/2312.10794v5#S3.Ex3 "3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) therefore indicates that $\mathsf{E}_{\beta}$ increases along trajectories of ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). (Similarly, should $V=-I_{d}$ , the energy $\mathsf{E}_{\beta}$ would decrease along trajectories.) This begs the question of characterizing the global minima and maxima of $\mathsf{E}_{\beta}$ , which is the goal of the following result.

###### Proposition 3.4.

Let $\beta>0$ and $d\geq 2$ . The unique global minimizer of $\mathsf{E}_{\beta}$ over $\mathcal{P}(\mathbb{S}^{d-1})$ is the uniform measure <sup>10</sup> <sup>10</sup> 10 That is, the Lebesgue measure on $\mathbb{S}^{d-1}$ , normalized to be a probability measure. $\sigma_{d}$ . Any global maximizer of $\mathsf{E}_{\beta}$ over $\mathcal{P}(\mathbb{S}^{d-1})$ is a Dirac mass $\delta_{x^{*}}$ centered at some point $x^{*}\in\mathbb{S}^{d-1}$ .

This result lends credence to our nomenclature of the case $V=I_{d}$ as *attractive*, and $V=-I_{d}$ as *repulsive*. The reader should be wary however that in this result we are minimizing or maximizing $\mathsf{E}_{\beta}$ among *all* probability measures on $\mathbb{S}^{d-1}$ . Should one focus solely on discrete measures, many global minima appear—these are discussed in Section [9.1](https://arxiv.org/html/2312.10794v5#S9.SS1 "9.1. The repulsive case ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") —. This is one point where the particle dynamics and the mean-field flow deviate. We now provide a brief proof of Proposition  [3.4](https://arxiv.org/html/2312.10794v5#S3.Thmtheorem4 "Proposition 3.4. ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers") (see \[[Tan17](https://arxiv.org/html/2312.10794v5#bib.bibx111)\] for a different approach).

###### Proof of.

The fact that any global maximizer is a Dirac mass is easy to see. We proceed with proving the rest of the statement. Let $f(t)=e^{\beta t}$ . The interaction energy then reads

|  | $$ \mathsf{E}_{\beta}[\mu]=\frac{1}{2}\iint f(\langle x,x^{\prime}\rangle)\,\mathrm{d}\mu(x)\,\mathrm{d}\mu(x^{\prime}). $$ |  |
| --- | --- | --- |

The proof relies on an ultraspherical (or Gegenbauer) polynomial expansion of $f(t)$ :

|  | $$ f(t)=\sum_{k=0}^{+\infty}\widehat{f}(k;\lambda)\frac{k+\lambda}{\lambda}C_{k}^{\lambda}(t) $$ |  |
| --- | --- | --- |

for $t\in[-1,1]$ , where $\lambda=\frac{d-2}{2}$ , $C_{k}^{\lambda}$ are Gegenbauer polynomials, and

|  | $$ \widehat{f}(k;\lambda)=\frac{\Gamma(\lambda+1)}{\Gamma(\lambda+\frac{1}{2})\Gamma(\frac{1}{2})}\frac{1}{C_{k}^{\lambda}(1)}\int_{-1}^{1}f(t)C^{\lambda}_{k}(t)(1-t^{2})^{\lambda-\frac{1}{2}}\,\mathrm{d}t $$ |  |
| --- | --- | --- |

where $C_{k}^{\lambda}(1)>0$ (see \[[DX13](https://arxiv.org/html/2312.10794v5#bib.bibx46), Section 1.2\]). According to \[[BD19](https://arxiv.org/html/2312.10794v5#bib.bibx14), Proposition 2.2\], a necessary and sufficient condition for [Proposition ˜ 3.4](https://arxiv.org/html/2312.10794v5#S3.Thmtheorem4 "Proposition 3.4. ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers") to hold is to ensure that $\widehat{f}(k;\lambda)>0$ for all $k\geq 1$ . To show this, we use the Rodrigues formula \[[Sze39](https://arxiv.org/html/2312.10794v5#bib.bibx109), 4.1.72\]

|  | $$ C_{k}^{\lambda}(t)=\frac{(-1)^{k}2^{k}}{k!}\frac{\Gamma(k+\lambda)\Gamma(k+2\lambda)}{\Gamma(\lambda)\Gamma(2k+2\lambda)}(1-t^{2})^{-(\lambda-\frac{1}{2})}\left(\frac{\,\mathrm{d}}{\,\mathrm{d}t}\right)^{k}(1-t^{2})^{k+\lambda-\frac{1}{2}}, $$ |  |
| --- | --- | --- |

and the fact that $C_{k}^{\lambda}(-t)=(-1)^{k}C_{k}^{\lambda}(t)$ for $t\in[-1,1]$ , which in combination with integration by parts yield

|  | $$ \int_{-1}^{1}t^{\ell}C_{k}^{\lambda}(t)(1-t^{2})^{\lambda-\frac{1}{2}}\,\mathrm{d}t\begin{cases}>0&\text{ if }\ell\geq k\text{ and }\ell-k\text{ is even}\\ =0&\text{ otherwise }.\end{cases} $$ |  |
| --- | --- | --- |

We conclude by using the power series expansion of $f$ . ∎

#### 3.3. A Wasserstein gradient flow proxy

In view of ([3.2](https://arxiv.org/html/2312.10794v5#S3.Ex3 "3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), one could hope to see the continuity equation ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) as the *Wasserstein gradient flow* of $\mathsf{E}_{\beta}$ , or possibly some other functional (see the seminal papers \[[Ott01](https://arxiv.org/html/2312.10794v5#bib.bibx97), [JKO98](https://arxiv.org/html/2312.10794v5#bib.bibx70)\], and \[[AGS05](https://arxiv.org/html/2312.10794v5#bib.bibx6), [Vil09](https://arxiv.org/html/2312.10794v5#bib.bibx120)\] for a complete treatment). The long-time asymptotics of the PDE can then be analyzed by studying convexity properties of the underlying functional, by analogy with gradient flows in the Euclidean case.

For ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) to be the Wasserstein gradient flow of $\mathsf{E}_{\beta}$ , the vector field $\mathcal{X}[\mu]$ defined in ([3.2](https://arxiv.org/html/2312.10794v5#S3.E2 "Equation 3.2 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) ought to be the gradient of the first variation $\delta\mathsf{E}_{\beta}$ of $\mathsf{E}_{\beta}$ . However, notice that $\mathcal{X}[\mu]$ is a logarithmic derivative:

<table><tbody><tr><td rowspan="1"><span>(3.7)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mrow><mo>∇</mo> <mi>log</mi></mrow> <mo></mo><mrow><mo>∫</mo> <mrow><msup><mi>β</mi> <mrow><mo>−</mo> <mn>1</mn></mrow></msup> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><mi>y</mi> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>y</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\mathcal{X}[\mu](x)=\nabla\log\int\beta^{-1}e^{\beta\langle x,y\rangle}\,\mathrm{d}\mu(y).</annotation></semantics></math></td><td></td></tr></tbody></table>

(This observation goes beyond $Q=K=I_{d}$ and $V=\pm I_{d}$ insofar as $Q^{\top}K=K^{\top}Q=\pm V$ ; see \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103), Assumption 1\]. Because of the lack of symmetry, it has been shown in \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\] that ([3.7](https://arxiv.org/html/2312.10794v5#S3.E7 "Equation 3.7 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is not the gradient of the first variation of a functional.

To overcome this limitation on $\mathbb{R}^{d}$ , thus without layer normalization, \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\] propose two ways to "symmetrize" ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) that both lead to a Wasserstein gradient flow; see \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103), Proposition 2\]. We focus here on the simplest one which consists in removing the logarithm in ([3.7](https://arxiv.org/html/2312.10794v5#S3.E7 "Equation 3.7 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), or equivalently to removing the denominator in ([3.2](https://arxiv.org/html/2312.10794v5#S3.E2 "Equation 3.2 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). This is one point where working on the unit sphere is useful: otherwise, the equation on $\mathbb{R}^{d}$ without layer normalization (as considered in \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\]) is ill-posed for general choices of matrices $V$ , due to the fact that the magnitude of the vector field $\mathcal{X}[\mu]$ grows exponentially with the size of the support of $\mu$ . On the contrary, on $\mathbb{S}^{d-1}$ the resulting equation is perfectly well-posed.

###### Remark 3.5 (Doubly stochastic kernel).

Considering the Transformer dynamics on $\mathbb{R}^{d}$ , thus without layer normalization, the authors in \[[SABP22](https://arxiv.org/html/2312.10794v5#bib.bibx103)\] propose an alternative symmetric model: they replace the self-attention (stochastic) matrix by a doubly stochastic one, generated from the Sinkhorn iteration. This leads to a Wasserstein gradient flow, whereby the resulting attention mechanism is implicitly expressed as a limit of Sinkhorn iterations. Understanding the emergence of clusters for this model is an interesting but possibly challenging question.

In view of the above discussion, we are inclined to propose the surrogate model

<table><tbody><tr><td rowspan="1"><span>(USA)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mn>1</mn> <mi>n</mi></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\frac{1}{n}\sum_{j=1}^{n}e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}x_{j}(t)\right),</annotation></semantics></math></td><td></td></tr></tbody></table>

which is obtained by replacing the partition function $Z_{\beta,i}(t)$ by $n$ . As a matter of fact, ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) presents a remarkably similar qualitative behavior—all of the results we show in this paper are essentially the same for both dynamics—.

The continuity equation corresponding to ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), namely

<table><tbody><tr><td rowspan="1"><span>(3.8)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow> <mo>+</mo> <mrow><mi>div</mi> <mo></mo><mrow><mo>(</mo><mrow><msubsup><mi>𝗣</mi> <mi>x</mi> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mo>∫</mo> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mi>x</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><msup><mi>x</mi> <mo>′</mo></msup> <mo></mo><mrow><mo>d</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><msup><mi>x</mi> <mo>′</mo></msup><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow> <mo></mo><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow></mtd></mtr> <mtr><mtd><mrow><msub><mi>μ</mi> <mrow><mo>|</mo> <mi>t</mi> <mo>=</mo> <mn>0</mn></mrow></msub> <mo>=</mo> <msub><mi>μ</mi> <mn>0</mn></msub></mrow></mtd></mtr></mtable></mrow> <annotation>\begin{dcases}\partial_{t}\mu(t,x)+\mathrm{div}\left(\bm{\mathsf{P}}^{\perp}_{x}\left(\int e^{\beta\langle x,x^{\prime}\rangle}x^{\prime}\,\mathrm{d}\mu(t,x^{\prime})\right)\mu(t,x)\right)=0&amp;\\ \mu_{|t=0}=\mu_{0}\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

for $(t,x)\in\mathbb{R}_{\geq 0}\times\mathbb{S}^{d-1}$ , can now be seen as a Wasserstein gradient flow for the interaction energy $\mathsf{E}_{\beta}$ defined in ([3.5](https://arxiv.org/html/2312.10794v5#S3.E5 "Equation 3.5 ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

###### Lemma 3.6.

Consider the interaction energy $\mathsf{E}_{\beta}:\mathcal{P}(\mathbb{S}^{d-1})\to\mathbb{R}_{\geq 0}$ defined in ([3.5](https://arxiv.org/html/2312.10794v5#S3.E5 "Equation 3.5 ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). Then the vector field

|  | $$ \mathcal{X}[\mu](x)=\bm{\mathsf{P}}^{\perp}_{x}\left(\int e^{\beta\langle x,x^{\prime}\rangle}x^{\prime}\,\mathrm{d}\mu(x^{\prime})\right) $$ |  |
| --- | --- | --- |

satisfies

<table><tbody><tr><td rowspan="1"><span>(3.9)</span></td><td></td><td><math><semantics><mrow><mrow><mi>𝒳</mi> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mrow><mo>∇</mo> <mrow><mi>δ</mi> <mo></mo><msub><mi>𝖤</mi> <mi>β</mi></msub></mrow></mrow> <mo></mo><mrow><mo>[</mo><mi>μ</mi><mo>]</mo></mrow> <mo></mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow> <annotation>\mathcal{X}[\mu](x)=\nabla\delta\mathsf{E}_{\beta}[\mu](x)</annotation></semantics></math></td><td></td></tr></tbody></table>

for any $\mu\in\mathcal{P}(\mathbb{S}^{d-1})$ and $x\in\mathbb{S}^{d-1}$ , where $\delta\mathsf{E}_{\beta}[\mu]$ denotes the first variation of $\mathsf{E}_{\beta}$ .

We omit the proof which follows from standard Otto calculus \[[Ott01](https://arxiv.org/html/2312.10794v5#bib.bibx97)\], \[[Vil09](https://arxiv.org/html/2312.10794v5#bib.bibx120), Chapter 15\], \[[CNWR24](https://arxiv.org/html/2312.10794v5#bib.bibx30), Chapter 5\]. We can actually write ([3.9](https://arxiv.org/html/2312.10794v5#S3.E9 "Equation 3.9 ‣ Lemma 3.6. ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) more succinctly by recalling the definition of the convolution of two functions on $\mathbb{S}^{d-1}$ \[[DX13](https://arxiv.org/html/2312.10794v5#bib.bibx46), Chapter 2\]: for any $g\in L^{1}(\mathbb{S}^{d-1})$ and $f:[-1,1]\to\mathbb{R}$ such that $t\mapsto(1-t^{2})^{\frac{d-3}{2}}f(t)$ is integrable,

|  | $$ (f\ast g)(x)=\int f(\langle x,y\rangle)g(y)\,\mathrm{d}\sigma_{d}(y). $$ |  |
| --- | --- | --- |

This definition has a natural extension to the convolution of a function $f$ (with the above integrability) and a measure $\mu\in\mathcal{P}(\mathbb{S}^{d-1})$ . We can hence rewrite

|  | $$ \mathsf{E}_{\beta}[\mu]=\frac{1}{2}\int(\mathsf{G}_{\beta}\ast\mu)(x)\,\mathrm{d}\mu(x) $$ |  |
| --- | --- | --- |

where $[-1,1]\ni\mathsf{G}_{\beta}(t)=\beta^{-1}e^{\beta t}$ , and so

|  | $$ \mathcal{X}[\mu](x)=\nabla(\mathsf{G}_{\beta}\ast\mu)(x). $$ |  |
| --- | --- | --- |

Thus, ([3.8](https://arxiv.org/html/2312.10794v5#S3.E8 "Equation 3.8 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) takes the equivalent form

<table><tbody><tr><td rowspan="1"><span>(3.10)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow> <mo>+</mo> <mrow><mi>div</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>∇</mo> <mrow><mo>(</mo><mrow><mrow><msub><mi>𝖦</mi> <mi>β</mi></msub> <mo>∗</mo> <mi>μ</mi></mrow> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mo>⋅</mo><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow> <mo>⁡</mo> <mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo></mo><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow></mtd> <mtd><mrow><mrow><mtext>for </mtext><mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>∈</mo> <mrow><msub><mi>ℝ</mi> <mrow><mo>≥</mo> <mn>0</mn></mrow></msub> <mo>×</mo> <msup><mi>𝕊</mi> <mrow><mi>d</mi> <mo>−</mo> <mn>1</mn></mrow></msup></mrow></mrow></mtd></mtr> <mtr><mtd><mrow><msub><mi>μ</mi> <mrow><mo>|</mo> <mi>t</mi> <mo>=</mo> <mn>0</mn></mrow></msub> <mo>=</mo> <msub><mi>μ</mi> <mn>0</mn></msub></mrow></mtd> <mtd><mrow><mrow><mrow><mtext>for </mtext><mo></mo><mi>x</mi></mrow> <mo>∈</mo> <msup><mi>𝕊</mi> <mrow><mi>d</mi> <mo>−</mo> <mn>1</mn></mrow></msup></mrow><mo>.</mo></mrow></mtd></mtr></mtable></mrow><annotation>\begin{dcases}\partial_{t}\mu(t,x)+\mathrm{div}\Big{(}\nabla\big{(}\mathsf{G}_{\beta}\ast\mu(t,\cdot)\big{)}(x)\mu(t,x)\Big{)}=0&amp;\text{ for }(t,x)\in\mathbb{R}_{\geq 0}\times\mathbb{S}^{d-1}\\ \mu_{|t=0}=\mu_{0}&amp;\text{ for }x\in\mathbb{S}^{d-1}.\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

The considerations above lead us to the following Lyapunov identity.

###### Lemma 3.7.

The solution $\mu\in C^{0}(\mathbb{R}_{\geq 0};\mathcal{P}(\mathbb{S}^{d-1}))$ to ([3.8](https://arxiv.org/html/2312.10794v5#S3.E8 "Equation 3.8 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) satisfies

|  | $$ \frac{\,\mathrm{d}}{\,\mathrm{d}t}\mathsf{E}_{\beta}[\mu(t)]=\int\left\\|\nabla\Big{(}\mathsf{G}_{\beta}\ast\mu(t,\cdot)\Big{)}(x)\right\\|^{2}\,\mathrm{d}\mu(t,x) $$ |  |
| --- | --- | --- | --- | --- |

for $t\geq 0$ .

Interestingly, ([3.10](https://arxiv.org/html/2312.10794v5#S3.E10 "Equation 3.10 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is an *aggregation* equation, versions of which have been studied in great depth in the literature. For instance, clustering in the spirit of an asymptotic collapse to a single Dirac measure located at the center of mass of the initial density $\mu(0,\cdot)$ has been shown for aggregation equations with singular kernels in \[[BCM08](https://arxiv.org/html/2312.10794v5#bib.bibx12), [BLR11](https://arxiv.org/html/2312.10794v5#bib.bibx18), [CDF <sup><span>+</span></sup> 11](https://arxiv.org/html/2312.10794v5#bib.bibx21)\], motivated by the Patlak-Keller-Segel model of chemotaxis. Here, one caveat (and subsequently, novelty) is that ([3.10](https://arxiv.org/html/2312.10794v5#S3.E10 "Equation 3.10 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is set on $\mathbb{S}^{d-1}$ which makes the analysis developed in these references difficult to adapt or replicate.

###### Remark 3.8 (Particle version).

Let us briefly sketch the particle version of the Wasserstein gradient flow ([3.8](https://arxiv.org/html/2312.10794v5#S3.E8 "Equation 3.8 ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). When $\mu(t)=\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}(t)}$ , the interaction energy ([3.5](https://arxiv.org/html/2312.10794v5#S3.E5 "Equation 3.5 ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) takes the form

|  | $$ \mathsf{E}_{\beta}(X)=\frac{1}{2\beta n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}e^{\beta\langle x_{i},x_{j}\rangle} $$ |  |
| --- | --- | --- |

where $X=(x_{1},\ldots,x_{n})\in(\mathbb{S}^{d-1})^{n}$ . Denoting by $\nabla_{X}$ the gradient associated to the standard Riemannian metric on $(\mathbb{S}^{d-1})^{n}$ , we get the dynamics

<table><tbody><tr><td rowspan="1"><span>(3.11)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mover><mi>X</mi> <mo>˙</mo></mover> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mi>n</mi> <mo></mo><mrow><msub><mo>∇</mo> <mi>X</mi></msub> <msub><mi>𝖤</mi> <mi>β</mi></msub></mrow> <mo></mo><mrow><mo>(</mo><mrow><mi>X</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\dot{X}(t)=n\nabla_{X}\mathsf{E}_{\beta}(X(t)).</annotation></semantics></math></td><td></td></tr></tbody></table>

Indeed, the gradient on $(\mathbb{S}^{d-1})^{n}$ is simply $\nabla=(\partial_{1},\ldots,\partial_{n})$ where $\partial_{i}$ is the gradient in $\mathbb{S}^{d-1}$ acting on the $i$ -th copy in $(\mathbb{S}^{d-1})^{n}$ . Therefore

|  | $$ {\partial_{i}\mathsf{E}_{\beta}}(X(t))=\frac{1}{\beta n^{2}}\sum_{j=1}^{n}{\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}}\left(e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}\beta x_{j}(t)\right)=\frac{1}{n}\dot{x}_{i}(t) $$ |  |
| --- | --- | --- |

which yields ([3.11](https://arxiv.org/html/2312.10794v5#S3.E11 "Equation 3.11 ‣ Remark 3.8 (Particle version). ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

Note that ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) also corresponds to a gradient flow of the same interaction energy albeit with respect to a Riemannian metric on the sphere different from the standard one (for $n=2$ the two are conformally equivalent). We provide more detail in the following section.

#### 3.4. () is a gradient flow for a modified metric

We will now briefly demonstrate that for a particular choice of parameters $(Q,K,V)$ , the true dynamics ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) can be seen as a gradient flow for $\mathsf{E}_{\beta}$ upon a modification of the metric on the tangent space of $(\mathbb{S}^{d-1})^{n}$ . This will facilitate qualitative analysis later on by using standard tools from dynamical systems. This insight can then be extrapolated to the corresponding continuity equation ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) as well, as seen in Section [3.4.2](https://arxiv.org/html/2312.10794v5#S3.SS4.SSS2 "3.4.2. The case of measures ‣ 3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers").

##### 3.4.1. The case of particles

We suppose that

|  | $$ Q^{\top}K\text{ is symmetric},\hskip 28.45274ptV=Q^{\top}K. $$ |  |
| --- | --- | --- |

We define a new metric on $(\mathbb{S}^{d-1})^{n}$ as follows. Let $X=(x_{1},\ldots,x_{n})\in(\mathbb{S}^{d-1})^{n}$ . Consider the inner product on $\mathrm{T}_{X}(\mathbb{S}^{d-1})^{n}$ given by

<table><tbody><tr><td rowspan="1"><span>(3.12)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mrow><mo>⟨</mo> <mrow><mo>(</mo><msub><mi>a</mi> <mn>1</mn></msub><mo>,</mo><mi>…</mi><mo>,</mo><msub><mi>a</mi> <mi>n</mi></msub><mo>)</mo></mrow><mo>,</mo><mrow><mo>(</mo><msub><mi>b</mi> <mn>1</mn></msub><mo>,</mo><mi>…</mi><mo>,</mo><msub><mi>b</mi> <mi>n</mi></msub><mo>)</mo></mrow> <mo>⟩</mo></mrow> <mi>X</mi></msub> <mo>=</mo> <mrow><munderover><mo>∑</mo> <mrow><mi>i</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>X</mi><mo>)</mo></mrow> <mo></mo><mrow><mo>⟨</mo> <msub><mi>a</mi> <mi>i</mi></msub><mo>,</mo><msub><mi>b</mi> <mi>i</mi></msub> <mo>⟩</mo></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\left\langle(a_{1},\ldots,a_{n}),(b_{1},\ldots,b_{n})\right\rangle_{X}=\sum_{i=1}^{n}Z_{\beta,i}(X)\langle a_{i},b_{i}\rangle\,,</annotation></semantics></math></td><td></td></tr></tbody></table>

where $a_{i},b_{i}\in\mathrm{T}_{x_{i}}\mathbb{S}^{d-1}$ , and

|  | $$ Z_{\beta,i}(X)=\sum_{j=1}^{n}e^{\beta\langle Vx_{i},x_{j}\rangle}. $$ |  |
| --- | --- | --- |

Set

|  | $$ \mathsf{E}_{\beta}(X)=\frac{1}{2\beta}\sum_{i=1}^{n}\sum_{j=1}^{n}e^{\beta\langle Vx_{i},x_{j}\rangle}. $$ |  |
| --- | --- | --- |

We now show that the dynamics ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) can be equivalently written as

|  | $$ \dot{X}(t)=\nabla\mathsf{E}_{\beta}(X(t)), $$ |  |
| --- | --- | --- |

where the gradient $\nabla$ is computed with respect to the metric ([3.12](https://arxiv.org/html/2312.10794v5#S3.E12 "Equation 3.12 ‣ 3.4.1. The case of particles ‣ 3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) on $(\mathbb{S}^{d-1})^{n}$ . To this end, we ought to show that for all vector fields $Y$ on $(\mathbb{S}^{d-1})^{n}$ and for all $X\in(\mathbb{S}^{d-1})^{n}$ ,

<table><tbody><tr><td rowspan="1"><span>(3.13)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mrow><mfrac><mi>d</mi> <mrow><mi>d</mi> <mo></mo><mi>t</mi></mrow></mfrac> <mo>|</mo></mrow> <mrow><mi>t</mi> <mo>=</mo> <mn>0</mn></mrow></msub> <mo></mo><msub><mi>𝖤</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mrow><msubsup><mi>Φ</mi> <mi>Y</mi> <mi>t</mi></msubsup> <mo></mo><mrow><mo>(</mo><mi>X</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow> <mo>=</mo> <msub><mrow><mo>⟨</mo> <mrow><mi>Y</mi> <mo></mo><mrow><mo>(</mo><mi>X</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>B</mi> <mo></mo><mrow><mo>(</mo><mi>X</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow> <mi>X</mi></msub></mrow> <annotation>\frac{\,\mathrm{d}}{\,\mathrm{d}t}\Big{|}_{t=0}\mathsf{E}_{\beta}\left(\Phi^{t}_{Y}(X)\right)=\langle Y(X),B(X)\rangle_{X}</annotation></semantics></math></td><td></td></tr></tbody></table>

holds, where $\Phi^{t}_{Y}$ is the flow associated to the vector field $Y$ , whereas $B=(B_{1},\ldots,B_{n})$ with

|  | $$ B_{i}=\bm{\mathsf{P}}^{\perp}_{x_{i}}\left(\frac{1}{Z_{\beta,i}(X)}\sum_{j=1}^{n}e^{\beta\langle Vx_{i},x_{j}\rangle}Vx_{j}\right)\in\mathrm{T}_{x_{i}}\mathbb{S}^{d-1}. $$ |  |
| --- | --- | --- |

By linearity, it is sufficient to show ([3.13](https://arxiv.org/html/2312.10794v5#S3.E13 "Equation 3.13 ‣ 3.4.1. The case of particles ‣ 3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) for vector fields $Y$ of the form

|  | $$ Y(X)=(Ax_{1},0,\ldots,0)\in\mathrm{T}_{X}(\mathbb{S}^{d-1})^{n}\, $$ |  |
| --- | --- | --- |

where $A$ is an arbitrary non-zero skew-symmetric matrix. Clearly

<table><tbody><tr><td rowspan="1"><span>(3.14)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msubsup><mi>Φ</mi> <mi>Y</mi> <mi>t</mi></msubsup> <mo></mo><mrow><mo>(</mo><mi>X</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mo>(</mo><mrow><msup><mi>e</mi> <mrow><mi>t</mi> <mo></mo><mi>A</mi></mrow></msup> <mo></mo><msub><mi>x</mi> <mn>1</mn></msub></mrow><mo>,</mo><msub><mi>x</mi> <mn>2</mn></msub><mo>,</mo><mi>…</mi><mo>,</mo><msub><mi>x</mi> <mi>n</mi></msub><mo>)</mo></mrow></mrow><mo>.</mo></mrow><annotation>\Phi_{Y}^{t}(X)=(e^{tA}x_{1},x_{2},\ldots,x_{n}).</annotation></semantics></math></td><td></td></tr></tbody></table>

One first computes

|  | $$ \frac{\,\mathrm{d}}{\,\mathrm{d}t}\Big{\|}_{t=0}\mathsf{E}_{\beta}\left(\Phi^{t}_{Y}(X)\right)=\sum_{j=1}^{n}e^{\beta\langle Vx_{1},x_{j}\rangle}\langle Ax_{1},Vx_{j}\rangle. $$ |  |
| --- | --- | --- | --- |

Now observe that $\langle Ax_{1},y\rangle=\langle Ax_{1},z\rangle$ for all skew-symmetric matrices $A$ if and only if $x_{1}(y^{\top}-z^{\top})$ is a symmetric matrix. Since $\bm{\mathsf{P}}^{\perp}_{x_{1}}=I_{d}-x_{1}x_{1}^{\top}$ , we see that

|  | $$ \sum_{j=1}^{n}e^{\beta\langle Vx_{1},x_{j}\rangle}\langle Ax_{1},Vx_{j}\rangle=\langle Y(X),B(X)\rangle_{X}, $$ |  |
| --- | --- | --- |

as desired.

##### 3.4.2. The case of measures

The above insight, which consists in reweighing the metric with respect to which the gradient is being taken, can be formally adapted to Wasserstein setting for ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))—which we recall, is not a gradient flow for the standard Wasserstein gradient of $\mathsf{E}_{\beta}$ —. But note that ([3.4](https://arxiv.org/html/2312.10794v5#S3.E4 "Equation 3.4 ‣ 3.1. The continuity equation ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) writes as

<table><tbody><tr><td rowspan="1"><span>(3.15)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>+</mo> <mrow><mi>div</mi> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mrow><mrow><mo>∇</mo> <mrow><mi>δ</mi> <mo></mo><msub><mi>𝖤</mi> <mi>β</mi></msub></mrow></mrow> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow></mrow> <mrow><mi>δ</mi> <mo></mo><msub><mi>𝖤</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>[</mo><mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow></mrow></mfrac> <mo></mo><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow><mo>,</mo></mrow><annotation>\partial_{t}\mu(t)+\mathrm{div}\left(\frac{\nabla\delta\mathsf{E}_{\beta}[\mu(t)]}{\delta\mathsf{E}_{\beta}[\mu(t)]}\mu(t)\right)=0,</annotation></semantics></math></td><td></td></tr></tbody></table>

with $\delta\mathsf{E}_{\beta}[\mu](x)=\int e^{\beta\langle x,y\rangle}\,\mathrm{d}\mu(y)$ . To avoid technicalities we henceforth only focus on the absolutely continuous case. For a fixed $\mu\in\mathcal{P}_{\mathrm{ac}}(\mathbb{S}^{d-1})$ , as in the well-known formal Riemannian reinterpretation of the Wasserstein space using Otto calculus \[[Ott01](https://arxiv.org/html/2312.10794v5#bib.bibx97)\], \[[CNWR24](https://arxiv.org/html/2312.10794v5#bib.bibx30), Chapter 5\], we consider

|  | $$ \mathrm{T}_{\mu}\mathcal{P}_{\mathrm{ac}}(\mathbb{S}^{d-1})=\overline{\left\{\nabla\psi\colon\psi\in C^{\infty}(\mathbb{S}^{d-1})\right\}}^{L^{2}(\mu)}, $$ |  |
| --- | --- | --- |

which, rather than endowing with the standard formal metric tensor given by the $\dot{H}^{1}$ –inner product

|  | $$ \langle\nabla\psi_{1},\nabla\psi_{2}\rangle_{\mu}:=\int\langle\nabla\psi_{1}(x),\nabla\psi_{2}(x)\rangle\,\mathrm{d}\mu(x), $$ |  |
| --- | --- | --- |

we instead endow with

|  | $$ \langle\nabla\psi_{1},\nabla\psi_{2}\rangle_{\mu,\mathsf{E}_{\beta}}:=\int\langle\nabla\psi_{1}(x),\nabla\psi_{2}(x)\rangle\,\delta\mathsf{E}_{\beta}[\mu](x)\,\mathrm{d}\mu(x). $$ |  |
| --- | --- | --- |

Continuing the Riemannian geometry analogy, through this metric tensor we can define a distance between $\mu_{0},\mu_{1}\in\mathcal{P}_{\mathrm{ac}}(\mathbb{S}^{d-1})$ by solving the variational problem

|  | $\displaystyle\inf_{(\mu(t),v(t))_{t\in[0,1]}}\Bigg{\{}\int_{0}^{1}\\|v(t)\\|^{2}_{\mu(t),\mathsf{E}_{\beta}}\,\mathrm{d}t$ | $\displaystyle\colon(\mu,v)\text{ satisfy }\eqref{eq: CE},\mu(0)=\mu_{0},\mu(1)=\mu_{1}\Bigg{\}},$ |  |
| --- | --- | --- | --- | --- | --- |

where

<table><tbody><tr><td rowspan="1"><span>(3.16)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow> <mo>+</mo> <mrow><mi>div</mi> <mo></mo><mrow><mo>(</mo><mrow><mi>v</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow> <mo></mo><mi>μ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mrow><mn>0</mn> <mrow><mrow><mtext>on </mtext><mo></mo><mrow><mo>[</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>]</mo></mrow></mrow> <mo>×</mo> <msup><mi>𝕊</mi> <mrow><mi>d</mi> <mo>−</mo> <mn>1</mn></mrow></msup></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\partial_{t}\mu(t,x)+\mathrm{div}(v(t,x)\mu(t,x))=0\hskip 28.45274pt\text{ on }[0,1]\times\mathbb{S}^{d-1}.</annotation></semantics></math></td><td></td></tr></tbody></table>

This variational problem is a generalization of the celebrated Benamou-Brenier formula \[[BB00](https://arxiv.org/html/2312.10794v5#bib.bibx10)\], the value of which we dub $W_{2,\mathsf{E}_{\beta}}^{2}(\mu_{0},\mu_{1})$ : it is a weighed Wasserstein distance. For a curve of measures $(\mu(t))_{t\geq 0}$ with tangent vectors $(v(t))_{t\geq 0}$ (meaning they solve ([3.16](https://arxiv.org/html/2312.10794v5#S3.E16 "Equation 3.16 ‣ 3.4.2. The case of measures ‣ 3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))), the Wasserstein gradient $\nabla\mkern-10.0mu\nabla_{\mathsf{E}_{\beta}}\mathsf{E}_{\beta}$ induced by this geometric setup is then the element of $\mathrm{T}_{\mu(t)}\mathcal{P}_{\mathrm{ac}}(\mathbb{S}^{d-1})$ such that

|  | $$ \partial_{t}\mathsf{E}_{\beta}[\mu(t)]=\langle\nabla\mkern-10.0mu\nabla_{\mathsf{E}_{\beta}}\mathsf{E}_{\beta}[\mu(t)],v(t)\rangle_{\mu(t),\mathsf{E}_{\beta}}. $$ |  |
| --- | --- | --- |

We can now demonstrate that the vector field driving ([3.15](https://arxiv.org/html/2312.10794v5#S3.E15 "Equation 3.15 ‣ 3.4.2. The case of measures ‣ 3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is the Wasserstein gradient of $\mathsf{E}_{\beta}$ corresponding to this geometric setup. Indeed as in \[[CNWR24](https://arxiv.org/html/2312.10794v5#bib.bibx30), Definition 5.9\] we first have

|  | $$ \partial_{t}\mathsf{E}_{\beta}[\mu(t)]=\int\delta\mathsf{E}_{\beta}[\mu(t)]\,\mathrm{d}\partial_{t}\mu(t). $$ |  |
| --- | --- | --- |

We then find

|  | $$ \int\delta\mathsf{E}_{\beta}[\mu(t)]\,\mathrm{d}\partial_{t}\mu(t)=\int\langle\nabla\delta\mathsf{E}_{\beta}[\mu(t)],v(t)\rangle\,\mathrm{d}\mu(t)=\left\langle\frac{\nabla\delta\mathsf{E}_{\beta}[\mu(t)]}{\delta\mathsf{E}_{\beta}[\mu(t)]},v(t)\right\rangle_{\mu(t),\mathsf{E}_{\beta}}, $$ |  |
| --- | --- | --- |

as desired. The literature studying weighed Wasserstein distances such as the one above is rather scarce, but a relevant reference is \[[Li21](https://arxiv.org/html/2312.10794v5#bib.bibx81)\].

## Part II Clustering

<svg xmlns="http://www.w3.org/2000/svg" class="ltx_picture ltx_centering" height="255.41" id="S3.F2.pic1" overflow="visible" version="1.1" viewBox="0 0 444.39 255.41" width="444.39"><g fill="#000000" stroke="#000000" transform="translate(0,255.41) matrix(1 0 0 -1 0 0) translate(-37.83,0) translate(0,33.04)"><g stroke-width="0.4pt"><g fill="#FF0000" fill-opacity="0.65" stroke-opacity="0.65"><path d="M 94.49 40.16 M 94.49 40.16 L 94.49 177.16 L 118.11 177.16 L 118.11 40.16 Z M 118.11 177.16" style="stroke:none"></path></g><g fill="#00FF00" fill-opacity="0.4" stroke-opacity="0.4"><path d="M 94.49 0 M 94.49 0 L 94.49 40.16 L 118.11 40.16 L 118.11 0 Z M 118.11 40.16" style="stroke:none"></path></g><g fill="#00FF00" fill-opacity="0.4" stroke-opacity="0.4"><path d="M 94.49 177.16 M 94.49 177.16 L 94.49 212.6 L 118.11 212.6 L 118.11 177.16 Z M 118.11 212.6" style="stroke:none"></path></g><g fill="#00FF00" fill-opacity="0.4" stroke-opacity="0.4"><path d="M 118.11 0 L 472.44 0 L 472.44 212.6 L 118.11 212.6 Z" style="stroke:none"></path></g><path d="M 94.49 4.72 L 94.49 -4.72" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 91.03 -18.53)"><foreignObject height="8.92" overflow="visible" style="--fo_width :0.5em;--fo_height:0.64em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 8.92)" width="6.92"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="2"><semantics><mn>2</mn> <annotation>2</annotation></semantics></math></span></span></foreignObject></g> <path d="M 118.11 4.72 L 118.11 -4.72" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 114.65 -18.53)"><foreignObject height="8.92" overflow="visible" style="--fo_width :0.5em;--fo_height:0.64em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 8.92)" width="6.92"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="3"><semantics><mn>3</mn> <annotation>3</annotation></semantics></math></span></span></foreignObject></g> <path d="M 236.22 4.72 L 236.22 -4.72" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 232.07 -15.57)"><foreignObject height="5.96" overflow="visible" style="--fo_width :0.6em;--fo_height:0.43em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 5.96)" width="8.31"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="n"><semantics><mi>n</mi> <annotation>n</annotation></semantics></math></span></span></foreignObject></g> <path d="M 425.2 4.72 L 425.2 -4.72" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 418.41 -19.22)"><foreignObject height="9.61" overflow="visible" style="--fo_width :0.98em;--fo_height:0.69em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 9.61)" width="13.56"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="d^{*}"><semantics><msup><mi>d</mi> <mo>∗</mo></msup> <annotation>d^{*}</annotation></semantics></math></span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 279.86 -28.43)"><foreignObject height="9.61" overflow="visible" style="--fo_width :0.52em;--fo_height:0.69em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 9.61)" width="7.2"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="d"><semantics><mi>d</mi> <annotation>d</annotation></semantics></math></span></span></foreignObject></g> <path d="M 92.13 40.16 L 99.21 40.16" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 81.5 35.7)"><foreignObject height="8.92" overflow="visible" style="--fo_width :0.93em;--fo_height:0.64em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 8.92)" width="12.82"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="1\hskip 4.26773pt"><semantics><mn>1</mn> <annotation>1\hskip 4.26773pt</annotation></semantics></math></span></span></foreignObject></g> <path d="M 92.13 177.16 L 99.21 177.16" style="fill:none"></path><g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 76.18 172.85)"><foreignObject height="17.27" overflow="visible" style="--fo_width :1.31em;--fo_height:0.94em;--fo_depth :0.31em;" transform="matrix(1 0 0 -1 0 12.95)" width="18.15"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="\frac{n^{2}}{\pi^{2}}\hskip 4.26773pt"><semantics><mfrac><msup><mi>n</mi> <mn>2</mn></msup> <msup><mi>π</mi> <mn>2</mn></msup></mfrac> <annotation>\frac{n^{2}}{\pi^{2}}\hskip 4.26773pt</annotation></semantics></math></span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 64.59 91.03)"><foreignObject height="12.3" overflow="visible" style="--fo_width :0.57em;--fo_height:0.69em;--fo_depth :0.19em;" transform="matrix(1 0 0 -1 0 9.61)" width="7.83"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="\beta"><semantics><mi>β</mi> <annotation>\beta</annotation></semantics></math></span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(0.0 1.0 -1.0 0.0 52.05 -3.91)"><foreignObject height="9.61" overflow="visible" style="--fo_width :1.59em;--fo_height:0.69em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 9.61)" width="21.99"><span xmlns="http://www.w3.org/1999/xhtml"><span>fast</span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(0.0 1.0 -1.0 0.0 52.14 21.43)"><foreignObject height="5.08" overflow="visible" style="--fo_width :1em;--fo_height:0.37em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 5.08)" width="13.84"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="\leftarrow"><semantics><mo stretchy="false">←</mo> <annotation>\leftarrow</annotation></semantics></math></span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(0.0 1.0 -1.0 0.0 54.55 53.23)"><foreignObject height="9.89" overflow="visible" style="--fo_width :0.5em;--fo_height:0.71em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 9.89)" width="6.92"><span xmlns="http://www.w3.org/1999/xhtml"><span><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="\rightarrow"><semantics><mo stretchy="false">→</mo> <annotation>\rightarrow</annotation></semantics></math></span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(0.0 1.0 -1.0 0.0 52.05 67.4)"><foreignObject height="9.61" overflow="visible" style="--fo_width :1.87em;--fo_height:0.69em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 9.61)" width="25.83"><span xmlns="http://www.w3.org/1999/xhtml"><span>slow</span></span></foreignObject></g> <g fill="#000000" stroke="#000000" transform="matrix(0.0 1.0 -1.0 0.0 108.49 52.02)"><foreignObject height="7.69" overflow="visible" style="--fo_width :9.23em;--fo_height:0.65em;--fo_depth :0em;" transform="matrix(1 0 0 -1 0 7.69)" width="108.56"><span xmlns="http://www.w3.org/1999/xhtml"><span><span>no result when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" data-latex="d=2"><semantics><mrow><mi>d</mi> <mo>=</mo> <mn>2</mn></mrow> <annotation>d=2</annotation></semantics></math></span></span></span></foreignObject></g></g><g stroke-dasharray="3.0pt,3.0pt" stroke-dashoffset="0.0pt" stroke-width="0.8pt"><path d="M 94.49 0 L 94.49 212.6" style="fill:none"></path></g><g stroke-dasharray="3.0pt,3.0pt" stroke-dashoffset="0.0pt" stroke-width="0.8pt"><path d="M 118.11 0 L 118.11 212.6" style="fill:none"></path></g><g stroke-dasharray="3.0pt,3.0pt" stroke-dashoffset="0.0pt" stroke-width="0.8pt"><path d="M 236.22 0 L 236.22 212.6" style="fill:none"></path></g><g stroke-dasharray="3.0pt,3.0pt" stroke-dashoffset="0.0pt" stroke-width="0.8pt"><path d="M 425.2 0 L 425.2 212.6" style="fill:none"></path></g><g stroke-width="0.8pt"><path d="M 94.49 0 L 471.46 0" style="fill:none"></path></g><g stroke-dasharray="none" stroke-dashoffset="0.0pt" stroke-linecap="round" stroke-linejoin="round" stroke-width="0.64pt" transform="matrix(1.0 0.0 0.0 1.0 471.46 0)"><path d="M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88" style="fill:none"></path></g><g stroke-width="0.8pt"><path d="M 94.49 -4.72 L 94.49 211.62" style="fill:none"></path></g><g stroke-dasharray="none" stroke-dashoffset="0.0pt" stroke-linecap="round" stroke-linejoin="round" stroke-width="0.64pt" transform="matrix(0.0 1.0 -1.0 0.0 94.49 211.62)"><path d="M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88" style="fill:none"></path></g></g></svg>

Figure 2. Green zones indicate regimes where convergence to a single cluster as $t\to+\infty$ can be proven. Here $n\geq 2$ is fixed. When $d$ is larger than specific thresholds, the long-time asymptotics can be chiseled out in finer detail. Convergence is slow when $\beta\gg 1$ (relative to the size of $d,n$ ), as even the exponential decay constant when $d\geq n$ is of the form $\lambda=O(e^{-\beta})$ and thus degenerates. One rather expects dynamic metastability. Section [4](https://arxiv.org/html/2312.10794v5#S4 "4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") addresses the case where $\beta$ is small and Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") where it’s large, whereas Section [6](https://arxiv.org/html/2312.10794v5#S6 "6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") covers the high-dimensional case at arbitrary $\beta$ .

As alluded to in the introductory discussion, clustering is of particular relevance in tasks such as sentiment analysis, masked language modeling, summarization, and so on. Therein, the output measure encodes the probability distribution of the missing tokens for instance, and its clustering indicates a small number of possible outcomes. In Sections [4](https://arxiv.org/html/2312.10794v5#S4 "4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), and [6](https://arxiv.org/html/2312.10794v5#S6 "6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), we show several results (mostly summarized in Figure [2](https://arxiv.org/html/2312.10794v5#S3.F2 "Figure 2 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) which indicate that the limiting distribution is a point mass. While it may appear that this leaves no room for diversity or randomness, which is at odds with practical observations, these results hold for the specific choice of parameter matrices, and apply in possibly very long-time horizons. Numerical experiments indicate a more subtle picture for different parameters—for instance, there is an appearance of a long metastable phase during which the particles coalesce in a small number of clusters, which seems consistent with behavior in trained models (Figure  [1](https://arxiv.org/html/2312.10794v5#S2.F1 "Figure 1 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers") in Section  [6.3](https://arxiv.org/html/2312.10794v5#S6.SS3 "6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"))—. We are not able to theoretically explain this behavior as of now.

Ultimately, the appearance of clusters is somewhat natural since the Transformer dynamics are a weighted average of all particles, with the weights being hard-wired to perform a fast selection of particles most similar to the $i$ -th particle being queried. This causes the emergence of leaders which attract all particles in their vicinity. In the natural language processing interpretation, where particles represent tokens, this further elucidates the wording *attention* as the mechanism of inter-token attraction, and the amplitude of the inner product between tokens can be seen as a measure of their *semantic similarity*.

### 4\. A single cluster for small β\\beta

As seen in Figure [2](https://arxiv.org/html/2312.10794v5#S3.F2 "Figure 2 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), the dimension $d$ and inverse temperature $\beta$ appear to play a key role in the clustering results we obtain. In this section and in Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), we begin by focusing on extreme choices of $\beta$ whilst $d,n$ are fixed. We first focus on the case $\beta=0$ (Section [4.1](https://arxiv.org/html/2312.10794v5#S4.SS1 "4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")), before moving to the case $\beta\ll 1$ by a perturbation argument (Section [4.2](https://arxiv.org/html/2312.10794v5#S4.SS2 "4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")). We cover the case where $\beta$ is sufficiently large, but finite, in Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"). The case $\beta=+\infty$ is of little interest since all particles are fixed by the evolution.

#### 4.1. The case β=0\\beta=0

For $\beta=0$ , both ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) and ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) read as

<table><tbody><tr><td rowspan="1"><span>(4.1)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msubsup><mi>𝗣</mi> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟂</mo></msubsup> <mo></mo><mrow><mo>(</mo><mrow><mfrac><mn>1</mn> <mi>n</mi></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow><mo>,</mo><mrow><mi>t</mi> <mo>⩾</mo> <mn>0</mn></mrow></mrow><mo>.</mo></mrow><annotation>\dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\frac{1}{n}\sum_{j=1}^{n}x_{j}(t)\right),\hskip 28.45274ptt\geqslant 0.</annotation></semantics></math></td><td></td></tr></tbody></table>

The following result shows that generically over the initial points, a single cluster emerges. It complements a known convergence result (\[[FL19](https://arxiv.org/html/2312.10794v5#bib.bibx53), Theorem 2\]) for ([4.1](https://arxiv.org/html/2312.10794v5#S4.E1 "Equation 4.1 ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")). In \[[FL19](https://arxiv.org/html/2312.10794v5#bib.bibx53), Theorem 2\], the authors show convergence to an antipodal configuration, in the sense that $n-1$ particles converge to some $x^{*}\in\mathbb{S}^{d-1}$ , with the last particle converging to $-x^{*}$ . Moreover, once convergence is shown to hold, it holds with an exponential rate. Mimicking the proof strategy of \[[BCM15](https://arxiv.org/html/2312.10794v5#bib.bibx13), Theorem 2.2\] and \[[HKR18](https://arxiv.org/html/2312.10794v5#bib.bibx65), Theorem 3.2\], we sharpen this result by showing that the appearance of an antipodal particle is non-generic over the choice of initial conditions.

###### Theorem 4.1.

Let $d,n\geq 2$ . For Lebesgue almost any initial sequence $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ , there exists some point $x^{*}\in\mathbb{S}^{d-1}$ such that the unique solution $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geq 0};(\mathbb{S}^{d-1})^{n})$ to the corresponding Cauchy problem for ([4.1](https://arxiv.org/html/2312.10794v5#S4.E1 "Equation 4.1 ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) satisfies

|  | $$ \lim_{t\to+\infty}x_{i}(t)=x^{*} $$ |  |
| --- | --- | --- |

for any $i\in[n]$ .

This is also referred to as convergence toward *consensus* in collective behavior models. We refer the interested reader to Appendix  [A](https://arxiv.org/html/2312.10794v5#A1 "Appendix A Proof of Theorem 4.1 ‣ Appendix ‣ A mathematical perspective on Transformers") for the proof, which relies on a gradient flow reinterpretation of ([4.1](https://arxiv.org/html/2312.10794v5#S4.E1 "Equation 4.1 ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")), much like the proofs of several subsequent theorems. We provide some comments in Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers").

#### 4.2. The case β≪1\\beta\\ll 1

Theorem [4.1](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem1 "Theorem 4.1. ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") has some implications for small but positive $\beta$ , something which is already seen in [Figure ˜ 3](https://arxiv.org/html/2312.10794v5#S6.F3 "In 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [Figure ˜ 4](https://arxiv.org/html/2312.10794v5#S6.F4 "In 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"). This is essentially due to the fact that, formally,

|  | $$ \dot{x}_{i}(t)=\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}\left(\frac{1}{n}\sum_{j=1}^{n}x_{j}(t)\right)+O(\beta) $$ |  |
| --- | --- | --- |

for $\beta\ll 1$ . So, during a time $\ll\beta^{-1}$ , the particles do not feel the influence of the remainder $O(\beta)$ and behave as in the regime $\beta=0$ . This motivates

###### Theorem 4.2.

Fix $d,n\geq 2$ . For $\beta\geq 0$ , let $\mathscr{S}_{\beta}\subset(\mathbb{S}^{d-1})^{n}$ be the subset consisting of all initial sequences for which the associated solution to the Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) (or ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))) converges to one cluster as $t\rightarrow+\infty$ . Then

|  | $$ \lim_{\beta\to 0}\mathbb{P}(\mathscr{S}_{\beta})=1. $$ |  |
| --- | --- | --- |

More generally, if $Q$ and $K$ are arbitrary $d\times d$ matrices, then the same result also holds for the Cauchy problem for ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with $V=I_{d}$ (or the natural analogue of ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with these parameters).

###### Proof.

We focus on the dynamics ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), but the proof is in fact identical in the case of ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

For $\alpha\in[0,1)$ , we say that a set formed from $n$ points $z_{1},\ldots,z_{n}\in(\mathbb{S}^{d-1})^{n}$ is *$\alpha$ –clustered* if for any $i,j\in[n]$ , $\langle z_{i},z_{j}\rangle>\alpha$ holds. Observe that if $\{z_{1},\ldots,z_{n}\}$ is $\alpha$ –clustered for some $\alpha\geq 0$ , then the solution to the Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) (for arbitrary $\beta\geq 0$ ) with this sequence as initial condition converges to a single cluster, since $w=z_{1}$ satisfies the assumption in [Lemma ˜ 6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers").

Now, for any integer $m\geq 1$ , we denote by $\mathscr{S}_{0}^{m}\subset\mathscr{S}_{0}$ the set of initial sequences $x_{1}(0),\ldots,x_{n}(0)$ in $(\mathbb{S}^{d-1})^{n}$ for which the solution $(x_{i}^{0}(\cdot))_{i\in[n]}$ to the associated Cauchy problem for ([4.1](https://arxiv.org/html/2312.10794v5#S4.E1 "Equation 4.1 ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) is $\frac{3}{4}$ –clustered at time $t=m$ , namely

<table><tbody><tr><td rowspan="1"><span>(4.2)</span></td><td></td><td><math><semantics><mrow><mrow><mo>⟨</mo> <mrow><msubsup><mi>x</mi> <mi>i</mi> <mn>0</mn></msubsup> <mo></mo><mrow><mo>(</mo><mi>m</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msubsup><mi>x</mi> <mi>j</mi> <mn>0</mn></msubsup> <mo></mo><mrow><mo>(</mo><mi>m</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow> <mo>&gt;</mo> <mfrac><mn>3</mn> <mn>4</mn></mfrac></mrow> <annotation>\langle x_{i}^{0}(m),x_{j}^{0}(m)\rangle&gt;\frac{3}{4}</annotation></semantics></math></td><td></td></tr></tbody></table>

holds for all $i,j\in[n]$ . We see that $\mathscr{S}_{0}^{m}$ is an open set for any integer $m\geq 1$ . Moreover, $\mathscr{S}_{0}^{m}\subset\mathscr{S}_{0}^{m+1}$ according to the proof of [Lemma ˜ 6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), and $\bigcup_{m=1}^{+\infty}\mathscr{S}_{0}^{m}=\mathscr{S}_{0}$ . This implies that

<table><tbody><tr><td rowspan="1"><span>(4.3)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><munder><mo>lim</mo> <mrow><mi>m</mi> <mo>→</mo> <mrow><mo>+</mo> <mi>∞</mi></mrow></mrow></munder> <mrow><mi>ℙ</mi> <mo></mo><mrow><mo>(</mo><msubsup><mi>𝒮</mi> <mn>0</mn> <mi>m</mi></msubsup><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>1</mn></mrow><mo>.</mo></mrow><annotation>\lim_{m\to+\infty}\mathbb{P}(\mathscr{S}_{0}^{m})=1.</annotation></semantics></math></td><td></td></tr></tbody></table>

We now show that the solution to ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is near that of ([4.1](https://arxiv.org/html/2312.10794v5#S4.E1 "Equation 4.1 ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")), starting from the same initial condition, when $\beta$ is small. Using the Duhamel formula, we find

|  | $\displaystyle x_{i}^{\beta}(t)-x_{i}^{0}(t)$ | $\displaystyle=\int_{0}^{t}\sum_{j=1}^{n}\left(\frac{e^{\beta\langle Qx_{i}^{\beta}(s),Kx_{j}^{\beta}(s)\rangle}}{\sum_{k=1}^{n}e^{\beta\langle Qx_{i}^{\beta}(s),Kx_{k}^{\beta}(s)\rangle}}\right)\bm{\mathsf{P}}^{\perp}_{x_{i}^{\beta}(s)}(x_{j}^{\beta}(s))\,\mathrm{d}s$ |  |
| --- | --- | --- | --- |
|  |  | $\displaystyle\quad-\int_{0}^{t}\frac{1}{n}\sum_{j=1}^{n}\bm{\mathsf{P}}^{\perp}_{x_{i}^{0}(s)}(x_{j}^{0}(s))\,\mathrm{d}s$ |  |
|  |  | $\displaystyle=\int_{0}^{t}\sum_{j=1}^{n}\left(\frac{1}{n}+O\left(\frac{\beta}{n}\right)\right)\bm{\mathsf{P}}^{\perp}_{x_{i}^{\beta}(s)}(x_{j}^{\beta}(s))\,\mathrm{d}s$ |  |
|  |  | $\displaystyle\quad-\int_{0}^{t}\frac{1}{n}\sum_{j=1}^{n}\bm{\mathsf{P}}^{\perp}_{x_{i}^{0}(s)}(x_{j}^{0}(s))\,\mathrm{d}s,$ |  |

where we used that all particles lie on $\mathbb{S}^{d-1}$ for all times. Employing Grönwall, we deduce

<table><tbody><tr><td rowspan="1"><span>(4.4)</span></td><td></td><td><math><semantics><mrow><mrow><mo>‖</mo> <mrow><mrow><msubsup><mi>x</mi> <mi>i</mi> <mi>β</mi></msubsup> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msubsup><mi>x</mi> <mi>i</mi> <mn>0</mn></msubsup> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>‖</mo></mrow> <mo>≤</mo> <mrow><mi>O</mi> <mo></mo><mrow><mo>(</mo><mi>β</mi><mo>)</mo></mrow> <mo></mo><msup><mi>e</mi> <mrow><mn>3</mn> <mo></mo><mi>t</mi></mrow></msup></mrow></mrow> <annotation>\displaystyle\left\|x_{i}^{\beta}(t)-x_{i}^{0}(t)\right\|\leq O(\beta)e^{3t}</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $t\geq 0$ , $\beta\geq 0$ and $i\in[n]$ . Due to ([4.4](https://arxiv.org/html/2312.10794v5#S4.E4 "Equation 4.4 ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")), there exists some $\beta_{m}>0$ such that for any $\beta\in[0,\beta_{m}]$ ,

<table><tbody><tr><td rowspan="1"><span>(4.5)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mo>‖</mo> <mrow><mrow><msubsup><mi>x</mi> <mi>i</mi> <mi>β</mi></msubsup> <mo></mo><mrow><mo>(</mo><mi>m</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msubsup><mi>x</mi> <mi>i</mi> <mn>0</mn></msubsup> <mo></mo><mrow><mo>(</mo><mi>m</mi><mo>)</mo></mrow></mrow></mrow> <mo>‖</mo></mrow> <mo>≤</mo> <mfrac><mn>1</mn> <mn>8</mn></mfrac></mrow><mo>.</mo></mrow><annotation>\left\|x_{i}^{\beta}(m)-x_{i}^{0}(m)\right\|\leq\frac{1}{8}.</annotation></semantics></math></td><td></td></tr></tbody></table>

For this to hold, we clearly need $\beta_{m}\to 0$ as $m\to+\infty$ . Combining ([4.2](https://arxiv.org/html/2312.10794v5#S4.E2 "Equation 4.2 ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) and ([4.5](https://arxiv.org/html/2312.10794v5#S4.E5 "Equation 4.5 ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")), we gather that for any initial condition in $\mathscr{S}_{0}^{m}$ , the solution $(x_{i}^{\beta}(\cdot))_{i\in[n]}$ to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is $\frac{1}{2}$ –clustered at time $t=m$ , namely satisfies

|  | $$ \langle x_{i}^{\beta}(m),x_{j}^{\beta}(m)\rangle>\frac{1}{2} $$ |  |
| --- | --- | --- |

for all $i,j\in[n]$ and $\beta\in[0,\beta_{m}]$ . Thus $\mathscr{S}_{0}^{m}\subset\mathscr{S}_{\beta}$ for any $\beta\in[0,\beta_{m}]$ by virtue of [Lemma ˜ 6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), which together with ([4.3](https://arxiv.org/html/2312.10794v5#S4.E3 "Equation 4.3 ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) concludes the proof. ∎

In the specific case where $Q^{\top}K=I_{d}$ , we can in fact significantly sharpen Theorem [4.2](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem2 "Theorem 4.2. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") by relying on the gradient flow structure evoked in Section [3.4](https://arxiv.org/html/2312.10794v5#S3.SS4 "3.4. (SA) is a gradient flow for a modified metric ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"). Namely, we can show the following.

###### Theorem 4.3.

Fix $d,n\geq 2$ . There exists a numerical constant $C>0$ such that whenever

|  | $$ \beta\leq Cn^{-1} $$ |  |
| --- | --- | --- |

the following holds. For Lebesgue almost any $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ , there exists $x^{*}\in\mathbb{S}^{d-1}$ such that the solution $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geq 0};(\mathbb{S}^{d-1})^{n})$ to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) (resp. for ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))) satisfies

|  | $$ \lim_{t\to+\infty}x_{i}(t)=x^{*} $$ |  |
| --- | --- | --- |

for all $i\in[n]$ .

Moreover, when $d=2$ one can take $\beta\leq 1$ and the same conclusion holds.

The proof can be found in Appendix [C](https://arxiv.org/html/2312.10794v5#A3 "Appendix C Proof of Theorem 4.3 ‣ Appendix ‣ A mathematical perspective on Transformers"). As for Theorem [4.1](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem1 "Theorem 4.1. ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), we briefly discuss the general strategy in Section [5](https://arxiv.org/html/2312.10794v5#S5 "5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") just below. The improvement to $\beta\leq 1$ when $d=2$ is due to the recent paper \[[CRMB24](https://arxiv.org/html/2312.10794v5#bib.bibx32)\].

### 5\. A single cluster for large β\\beta

The chain of thought leading to the proof of Theorem [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") also leads to

###### Theorem 5.1.

Fix $d,n\geq 2$ . There exists a constant $C=C(d)>0$ depending only on $d$ such that whenever

|  | $$ \beta\geq Cn^{2}, $$ |  |
| --- | --- | --- |

the conclusion of Theorem [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") holds for both ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) and ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

We refer the interested reader to Appendix [B](https://arxiv.org/html/2312.10794v5#A2 "Appendix B Proof of Theorem 5.1 ‣ Appendix ‣ A mathematical perspective on Transformers") for the proof, which, much like those of Theorem [4.1](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem1 "Theorem 4.1. ‣ 4.1. The case 𝛽=0 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") relies on the gradient flow interpretation of the dynamics. We give a general outline thereof. Since all the intervening functions and metrics are real-analytic, the celebrated *Łojasiewicz theorem* \[[Loj63](https://arxiv.org/html/2312.10794v5#bib.bibx86)\] implies that for any initial condition, the gradient flow converges to some critical point of $\mathsf{E}_{\beta}$ as $t\to+\infty$ . The genericity over the initial configurations seen in the statements comes from the *center-stable manifold theorem* (Lemma [A.1](https://arxiv.org/html/2312.10794v5#A1.Thmtheorem1 "Lemma A.1. ‣ Appendix A Proof of Theorem 4.1 ‣ Appendix ‣ A mathematical perspective on Transformers")). The former ensures that for almost every initial configuration, the gradient flow does not converge to a strict saddle point of $\mathsf{E}_{\beta}$ (namely critical points where the Hessian has at least one positive eigenvalue). Whence, generically over the initial configurations, the gradient flow converges to a local maximum. One is then left analyzing the landscape of the underlying energy, with the goal of ensuring that all local maxima are necessarily global.

### 6\. The high-dimensional case

We now elucidate the role that the dimension $d$ plays in clustering results. To start, it turns out that the restrictions on $\beta$ provided by Theorems [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [5.1](https://arxiv.org/html/2312.10794v5#S5.Thmtheorem1 "Theorem 5.1. ‣ 5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") are specific to the two-dimensional case. The following result is shown in \[[MTG17](https://arxiv.org/html/2312.10794v5#bib.bibx92), [CRMB24](https://arxiv.org/html/2312.10794v5#bib.bibx32)\].

###### Theorem 6.1 (\[, \]).

Fix $n\geq 2$ , $d\geq 3$ and $\beta\geq 0$ . Then the conclusion of Theorem [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") holds for both ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) and ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

In Section [6.1](https://arxiv.org/html/2312.10794v5#S6.SS1 "6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") we show that the convergence of Theorem [6.1](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem1 "Theorem 6.1 ([MTG17, CRMB24]). ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") is exponentially fast when $d\geq n$ (although, with a decay constant that is exponentially small in $\beta$ ) and in Section [6.2](https://arxiv.org/html/2312.10794v5#S6.SS2 "6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") we describe the full dynamics when $n$ is fixed and $d\to+\infty$ . We discuss some numerical experiments and posit questions on the intermediate behavior of the dynamics when $\beta\gg 1$ in Section [6.3](https://arxiv.org/html/2312.10794v5#S6.SS3 "6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers").

We were made aware of Theorem [6.1](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem1 "Theorem 6.1 ([MTG17, CRMB24]). ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") after the first version of this manuscript. The downside of our original proof, which caused us to miss the full range of $\beta$ , is that we focused on $d=2$ , where the natural perturbations to a critical point involve selecting which particles are moving and which are standing still. However, in $d>2$ , one can move all particles in the same direction (as is done in the proof in \[[CRMB24](https://arxiv.org/html/2312.10794v5#bib.bibx32)\]). Using that there is a continuum of directions and only finitely many points, one can find some direction that brings all of them closer.

###### Remark 6.2 (Invariant measures).

In the theory of dynamical systems, often the first question of interest is to find smooth invariant measures. It is clear that whenever conclusions of Theorem [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") hold (in particular, for all $\beta,n$ when $d\geq 3$ ), neither ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) nor ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) may possess a smooth invariant measure.

#### 6.1. Clustering at an exponential rate when d≥nd\\geq n

One can ask whether for almost every initial configuration, the convergence provided by all of the results above holds with some rate. The answer is affirmative—and the rate is in fact exponential—when the initial configuration lies in an open hemisphere.

###### Theorem 6.3.

Let $n\geq 1$ and $\beta>0$ . Suppose $d\geqslant n$ . Consider the unique solution $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geq 0};(\mathbb{S}^{d-1})^{n})$ to the Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) or ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), corresponding to an initial sequence of points $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ distributed uniformly at random. Then almost surely there exists $x^{*}\in\mathbb{S}^{d-1}$ and constants $C,\lambda>0$ such that

<table><tbody><tr><td rowspan="1"><span>(6.1)</span></td><td></td><td><math><semantics><mrow><mrow><mo>‖</mo> <mrow><mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <msup><mi>x</mi> <mo>∗</mo></msup></mrow> <mo>‖</mo></mrow> <mo>≤</mo> <mrow><mi>C</mi> <mo></mo><msup><mi>e</mi> <mrow><mo>−</mo> <mrow><mi>λ</mi> <mo></mo><mi>t</mi></mrow></mrow></msup></mrow></mrow> <annotation>\|x_{i}(t)-x^{*}\|\leq Ce^{-\lambda t}</annotation></semantics></math></td><td></td></tr></tbody></table>

holds for all $i\in[n]$ and $t\geq 0$ .

In fact, let $Q$ and $K$ be arbitrary $d\times d$ matrices. Then the same result also holds for the solution to the corresponding Cauchy problem for ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with $V=I_{d}$ (or the natural analogue of ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with these parameters).

When $d\geq n$ and the points $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ are distributed uniformly at random, with probability one there exists <sup>11</sup> <sup>11</sup> 11 This weak version of Wendel’s theorem (Theorem  [6.7](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem7 "Theorem 6.7 (Wendel, [Wen62]). ‣ Step 2. Exponential rate. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) is easy to see directly. $w\in\mathbb{S}^{d-1}$ such that $\langle w,x_{i}(0)\rangle>0$ for any $i\in[n]$ . In other words, all of the initial points lie in an open hemisphere almost surely. The proof of [Theorem ˜ 6.3](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem3 "Theorem 6.3. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") thus follows as a direct corollary of the following result, which holds for any $n\geq 1$ and $d\geq 2$ :

###### Lemma 6.4 (Cone collapse).

Let $\beta>0$ and let $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ be such that there exists $w\in\mathbb{S}^{d-1}$ for which $\langle x_{i}(0),w\rangle>0$ for any $i\in[n]$ . Consider the unique solution $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geq 0};(\mathbb{S}^{d-1})^{n})$ to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) or ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). Then there exists $x^{*}\in\mathbb{S}^{d-1}$ and constants $C,\lambda>0$ such that

|  | $$ \\|x_{i}(t)-x^{*}\\|\leq Ce^{-\lambda t} $$ |  |
| --- | --- | --- | --- | --- |

holds for all $i\in[n]$ and $t\geq 0$ .

In fact, let $Q$ and $K$ be arbitrary $d\times d$ matrices. Then the same result also holds for the solution to the corresponding Cauchy problem for ([2.3](https://arxiv.org/html/2312.10794v5#S2.E3 "Equation 2.3 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with $V=I_{d}$ (or the natural analogue of ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with these parameters).

###### Remark 6.5.

Lemma [6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") implies that $\left\{(\bar{x}_{i})_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}\colon\bar{x}_{1}=\ldots=\bar{x}_{n}\right\}$ is Lyapunov asymptotically stable as a set. In fact, it is exponentially stable.

Lemma [6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") is reminiscent of results on interacting particle systems on the sphere (see \[[CLP15](https://arxiv.org/html/2312.10794v5#bib.bibx27), Theorem 3.7\] for instance), and the literature on synchronization for the Kuramoto model on the circle (\[[ABK <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx1), Lemma 2.8\], \[[HR20](https://arxiv.org/html/2312.10794v5#bib.bibx67), Theorem 3.1\] and Section [7.2](https://arxiv.org/html/2312.10794v5#S7.SS2 "7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")). We often make use of the following elementary lemma.

###### Lemma 6.6.

Let $f:\mathbb{R}_{\geq 0}\to\mathbb{R}$ be a differentiable function such that

|  | $$ \int_{0}^{+\infty}\|f(t)\|\,\mathrm{d}t+\sup_{t\in\mathbb{R}_{\geq 0}}\left\|\dot{f}(t)\right\|<+\infty. $$ |  |
| --- | --- | --- | --- | --- | --- | --- |

Then $\lim_{t\to+\infty}f(t)=0$ .

The proof of [Lemma ˜ 6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") is an adaptation of \[[CLP15](https://arxiv.org/html/2312.10794v5#bib.bibx27), Theorem 1\]. We present it here for completeness.

###### Proof of.

We focus on the case ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), and set

|  | $$ a_{ij}(t):=n^{-1}e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}>0. $$ |  |
| --- | --- | --- |

The proof for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) is identical, and one only needs to change the coefficients $a_{ij}(t)$ by $Z_{\beta,i}(t)^{-1}e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}$ throughout. Also note that since we only make use of the positivity of the coefficients $a_{i,j}(t)$ throughout the proof, all arguments are readily generalizable to the case of arbitrary $d\times d$ matrices $Q$ and $K$ appearing in the inner products.

##### Step 1. Clustering

For $t\geq 0$ , consider

|  | $$ i(t)\in\operatorname*{arg\,min}_{i\in[n]}\langle x_{i}(t),w\rangle. $$ |  |
| --- | --- | --- |

Fix $t_{0}\geq 0$ . We have

|  | $\displaystyle\left(\frac{\,\mathrm{d}}{\,\mathrm{d}t}\langle x_{i(t_{0})}(\cdot),w\rangle\right)\Bigm{\|}_{t=t_{0}}$ |  |
| --- | --- | --- | --- |
|  | $\displaystyle=\sum_{j=1}^{n}a_{i(t_{0})j}(t_{0})\Big{(}\langle x_{j}(t_{0}),w\rangle-\langle x_{i(t_{0})}(t_{0}),x_{j}(t_{0})\rangle\langle x_{i(t_{0})}(t_{0}),w\rangle\Big{)}\geq 0.$ |  |

This implies that all points remain within the same open hemisphere at all times and the map

|  | $$ t\mapsto r(t):=\min_{i\in[n]}\langle x_{i}(t),w\rangle $$ |  |
| --- | --- | --- |

is non-decreasing on $\mathbb{R}_{\geq 0}$ . It is also bounded from above by $1$ . We may thus define $r_{\infty}:=\lim_{t\rightarrow+\infty}r(t)$ . Note that $r_{\infty}\geq r(0)>0$ by assumption. By compactness, there exist a sequence of times $\{t_{k}\}_{k=1}^{+\infty}$ with $t_{k}\rightarrow+\infty$ , and some $(\overline{x}_{i})_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ such that $\lim_{k\rightarrow+\infty}x_{i}(t_{k})=\overline{x}_{i}$ for all $i\in[n]$ . Using the definition of $r(t)$ , we also find that

|  | $$ \langle\overline{x}_{j},w\rangle\geq r_{\infty} $$ |  |
| --- | --- | --- |

for all $j\in[n]$ , and by continuity, there exists $i\in[n]$ such that $\langle\overline{x}_{i},w\rangle=r_{\infty}$ . Then

<table><tbody><tr><td rowspan="1"><span>(6.2)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><munder><mo>lim</mo> <mrow><mi>k</mi> <mo>→</mo> <mrow><mo>+</mo> <mi>∞</mi></mrow></mrow></munder> <mrow><mo>⟨</mo> <mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><msub><mi>t</mi> <mi>k</mi></msub><mo>)</mo></mrow></mrow><mo>,</mo><mi>w</mi> <mo>⟩</mo></mrow></mrow> <mo>=</mo> <mrow><mstyle><munderover><mo>∑</mo> <mtable><mtr><mtd><mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow></mtd></mtr></mtable> <mi>n</mi></munderover></mstyle> <mrow><msub><mover><mi>a</mi> <mo>¯</mo></mover> <mrow><mi>i</mi> <mo></mo><mi>j</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mrow><mrow><mo>⟨</mo> <mi>w</mi><mo>,</mo><msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>j</mi></msub> <mo>⟩</mo></mrow> <mo>−</mo> <mrow><mrow><mo>⟨</mo> <msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>i</mi></msub><mo>,</mo><msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>j</mi></msub> <mo>⟩</mo></mrow> <mo></mo><mrow><mo>⟨</mo> <msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>i</mi></msub><mo>,</mo><mi>w</mi> <mo>⟩</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>≥</mo> <mrow><msub><mi>r</mi> <mi>∞</mi></msub> <mo></mo><mrow><mstyle><munderover><mo>∑</mo> <mtable><mtr><mtd><mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow></mtd></mtr></mtable> <mi>n</mi></munderover></mstyle> <mrow><msub><mover><mi>a</mi> <mo>¯</mo></mover> <mrow><mi>i</mi> <mo></mo><mi>j</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mrow><mn>1</mn> <mo>−</mo> <mrow><mo>⟨</mo> <msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>i</mi></msub><mo>,</mo><msub><mover><mi>x</mi> <mo>¯</mo></mover> <mi>j</mi></msub> <mo>⟩</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\displaystyle\lim_{k\rightarrow+\infty}\langle\dot{x}_{i}(t_{k}),w\rangle=\sum_{\begin{subarray}{c}j=1\end{subarray}}^{n}\overline{a}_{ij}(\langle w,\overline{x}_{j}\rangle-\langle\overline{x}_{i},\overline{x}_{j}\rangle\langle\overline{x}_{i},w\rangle)\geq r_{\infty}\sum_{\begin{subarray}{c}j=1\end{subarray}}^{n}\overline{a}_{ij}(1-\langle\overline{x}_{i},\overline{x}_{j}\rangle),</annotation></semantics></math></td><td></td></tr></tbody></table>

where we set $\overline{a}_{ij}:=e^{\beta\langle\overline{x}_{i},\overline{x}_{j}\rangle}>0$ . Notice that

|  | $$ \lim_{k\to+\infty}\int_{t_{k}}^{+\infty}\langle\dot{x}_{i}(s),w\rangle\,\mathrm{d}s=r_{\infty}-\lim_{k\to+\infty}\langle x_{i}(t_{k}),w\rangle=0, $$ |  |
| --- | --- | --- |

and by using the equation ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) we also find that $|\langle\ddot{x}_{i}(t),w\rangle|=O(e^{2\beta})$ for any $t\geq 0$ . Therefore by Lemma  [6.6](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem6 "Lemma 6.6. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), the left-hand side of ([6.2](https://arxiv.org/html/2312.10794v5#S6.E2 "Equation 6.2 ‣ Step 1. Clustering ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) is equal to $0$ , and consequently the right-hand side term as well. This implies that $\overline{x}_{1}=\ldots=\overline{x}_{n}:=x^{*}$ . Repeating the argument by replacing $w$ with $x^{*}$ , we see that the extraction of a sequence $\{t_{k}\}_{k=1}^{+\infty}$ as above is not necessary, and therefore

<table><tbody><tr><td rowspan="1"><span>(6.3)</span></td><td></td><td><math><semantics><mrow><mrow><munder><mo>lim</mo> <mrow><mi>t</mi> <mo>→</mo> <mrow><mo>+</mo> <mi>∞</mi></mrow></mrow></munder> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <msup><mi>x</mi> <mo>∗</mo></msup></mrow> <annotation>\lim_{t\to+\infty}x_{i}(t)=x^{*}</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $i\in[n]$ .

##### Step 2. Exponential rate.

We now improve upon ([6.3](https://arxiv.org/html/2312.10794v5#S6.E3 "Equation 6.3 ‣ Step 1. Clustering ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")). Set

|  | $$ \alpha(t):=\min_{i\in[n]}\langle x_{i}(t),x^{*}\rangle. $$ |  |
| --- | --- | --- |

From ([6.3](https://arxiv.org/html/2312.10794v5#S6.E3 "Equation 6.3 ‣ Step 1. Clustering ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) we gather that there exists some $t_{0}>0$ such that $\alpha(t)\geq\frac{1}{2}$ for all $t\geq t_{0}$ . Also, in view of what precedes we know that $x^{*}$ lies in the convex cone generated by the points $x_{1}(t),\ldots,x_{n}(t)$ for any $t>0$ . Thus, there exists some $\eta\in(0,1]$ such that $\eta x^{*}$ is a convex combination of the points $x_{1}(t),\ldots,x_{n}(t)$ , which implies that

<table><tbody><tr><td rowspan="1"><span>(6.4)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msup><mi>x</mi> <mo>∗</mo></msup> <mo>=</mo> <mrow><mrow><munderover><mo>∑</mo> <mrow><mi>k</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>θ</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><msub><mi>x</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>,</mo><mtext> for some </mtext></mrow></mrow><mrow><mrow><mrow><munderover><mo>∑</mo> <mrow><mi>k</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>θ</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>≥</mo> <mn>1</mn></mrow><mo>,</mo><mrow><mrow><mrow><msub><mi>θ</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>≥</mo> <mn>0</mn></mrow> <mrow><mrow><mo>∀</mo> <mi>k</mi></mrow> <mo>∈</mo> <mrow><mo>[</mo><mi>n</mi><mo>]</mo></mrow></mrow></mrow></mrow></mrow><mo>.</mo></mrow><annotation>x^{*}=\sum_{k=1}^{n}\theta_{k}(t)x_{k}(t),\hskip 14.22636pt\text{ for some }\hskip 14.22636pt\sum_{k=1}^{n}\theta_{k}(t)\geq 1,\quad\theta_{k}(t)\geq 0\quad\forall k\in[n].</annotation></semantics></math></td><td></td></tr></tbody></table>

For any $t$ , we denote by $i(t)$ an element of $\operatorname*{arg\,min}(\langle x_{i}(t),x^{*}\rangle)$ for which $\langle\dot{x}_{i}(t),x^{*}\rangle$ is smallest. It follows from a Taylor expansion of $\langle x_{i}(t+h),x^{*}\rangle$ for $h>0$ and $i\in[n]$ that

|  | $$ \dot{\alpha}(t)=\langle\dot{x}_{i(t)}(t),x^{*}\rangle. $$ |  |
| --- | --- | --- |

Therefore

<table><tbody><tr><td rowspan="1"><span>(6.5)</span></td><td></td><td><math><semantics><mrow><mrow><mover><mi>α</mi> <mo>˙</mo></mover> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mo>⟨</mo> <mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><msup><mi>x</mi> <mo>∗</mo></msup> <mo>⟩</mo></mrow> <mo>≥</mo> <mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>a</mi> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mi>j</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mrow><mo>(</mo><mrow><mn>1</mn> <mo>−</mo> <mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow><mo>)</mo></mrow> <mo></mo><mi>α</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow> <annotation>\dot{\alpha}(t)=\langle\dot{x}_{i(t)}(t),x^{*}\rangle\geq\sum_{j=1}^{n}a_{i(t)j}(t)(1-\langle x_{i(t)}(t),x_{j}(t)\rangle)\alpha(t)</annotation></semantics></math></td><td></td></tr></tbody></table>

On another hand,

<table><tbody><tr><td rowspan="1"><span>(6.6)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><munder><mi>min</mi> <mrow><mi>j</mi> <mo>∈</mo> <mrow><mo>[</mo><mi>n</mi><mo>]</mo></mrow></mrow></munder> <mo>⁡</mo> <mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow> <mo>≤</mo> <mrow><munderover><mo>∑</mo> <mrow><mi>k</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msub><mi>θ</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow> <mo></mo><mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></mrow> <mo>=</mo> <mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mrow><mi>i</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><msup><mi>x</mi> <mo>∗</mo></msup> <mo>⟩</mo></mrow> <mo>=</mo> <mrow><mi>α</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\min_{j\in[n]}\langle x_{i(t)}(t),x_{j}(t)\rangle\leq\sum_{k=1}^{n}\theta_{k}(t)\langle x_{i(t)}(t),x_{k}(t)\rangle=\langle x_{i(t)}(t),x^{*}\rangle=\alpha(t).</annotation></semantics></math></td><td></td></tr></tbody></table>

Plugging ([6.6](https://arxiv.org/html/2312.10794v5#S6.E6 "Equation 6.6 ‣ Step 2. Exponential rate. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) into ([6.5](https://arxiv.org/html/2312.10794v5#S6.E5 "Equation 6.5 ‣ Step 2. Exponential rate. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) and using $a_{ij}(t)\geq n^{-1}e^{-2\beta}$ we get

<table><tbody><tr><td rowspan="1"><span>(6.7)</span></td><td></td><td><math><semantics><mrow><mrow><mover><mi>α</mi> <mo>˙</mo></mover> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>≥</mo> <mrow><mfrac><mn>1</mn> <mrow><mn>2</mn> <mo></mo><mi>n</mi> <mo></mo><msup><mi>e</mi> <mrow><mn>2</mn> <mo></mo><mi>β</mi></mrow></msup></mrow></mfrac> <mo></mo><mrow><mo>(</mo><mrow><mn>1</mn> <mo>−</mo> <mrow><mi>α</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <annotation>\dot{\alpha}(t)\geq\frac{1}{2ne^{2\beta}}(1-\alpha(t))</annotation></semantics></math></td><td></td></tr></tbody></table>

for $t\geq t_{0}$ . Applying the Grönwall inequality we get

<table><tbody><tr><td rowspan="1"><span>(6.8)</span></td><td></td><td><math><semantics><mrow><mrow><mn>1</mn> <mo>−</mo> <mrow><mi>α</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>≤</mo> <mrow><mfrac><mn>1</mn> <mn>2</mn></mfrac> <mo></mo><msup><mi>e</mi> <mrow><mo>−</mo> <mrow><mfrac><mn>1</mn> <mrow><mn>2</mn> <mo></mo><mi>n</mi> <mo></mo><msup><mi>e</mi> <mi>β</mi></msup></mrow></mfrac> <mo></mo><mrow><mo>(</mo><mrow><mi>t</mi> <mo>−</mo> <msub><mi>t</mi> <mn>0</mn></msub></mrow><mo>)</mo></mrow></mrow></mrow></msup></mrow></mrow> <annotation>1-\alpha(t)\leq\frac{1}{2}e^{-\frac{1}{2ne^{\beta}}(t-t_{0})}</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $t\geq t_{0}$ . The conclusion follows. ∎

In the case $d<n$ , we can still apply Wendel’s theorem (recalled below) together with Lemma  [6.4](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem4 "Lemma 6.4 (Cone collapse). ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") to obtain clustering to a single point with probability at least $p_{n,d}$ for some explicit $p_{n,d}\in(0,1)$ .

###### Theorem 6.7 (Wendel, \[\]).

Let $d,n\geq 1$ be such that $d\leq n$ . Let $x_{1},\ldots,x_{n}$ be $n$ i.i.d. uniformly distributed points on $\mathbb{S}^{d-1}$ . The probability that these points all lie in the same hemisphere is:

|  | $$ \mathbb{P}\Big{(}\exists w\in\mathbb{S}^{d-1}\colon\langle x_{i},w\rangle>0\quad\mathrm{for}\,\mathrm{all}\quad i\in[n]\Big{)}=2^{-(n-1)}\sum_{k=0}^{d-1}\binom{n-1}{k}. $$ |  |
| --- | --- | --- |

#### 6.2. More precise quantitative convergence

When $n$ is fixed and $d\rightarrow+\infty$ , in addition to showing the formation of a cluster as in Theorem  [6.3](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem3 "Theorem 6.3. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), it is possible to quantitatively describe the entire evolution of the particles with high probability. To motivate this, on the one hand we note that since the dynamics evolve on $\mathbb{S}^{d-1}$ , inner products are representative of the distance between points, and clustering occurs if $\langle x_{i}(t),x_{j}(t)\rangle\rightarrow 1$ for any $(i,j)\in[n]^{2}$ as $t\rightarrow+\infty$ . On the other hand, if $d\gg n$ , $n$ points in a generic initial sequence are *almost orthogonal* by concentration of measure \[[Ver18](https://arxiv.org/html/2312.10794v5#bib.bibx118), Chapter 3\], and we are thus able to compare their evolution with that of an initial sequence of truly *orthogonal* ones.

We begin by describing the case of exactly orthogonal initial particles, which is particularly simple as the dynamics are described by a single parameter.

###### Theorem 6.8.

Let $\beta\geq 0$ , $d,n\geq 2$ be arbitrary. Consider an initial sequence $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ of $n$ pairwise orthogonal points: $\langle x_{i}(0),x_{j}(0)\rangle=0$ for $i\neq j$ , and let $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geqslant 0};(\mathbb{S}^{d-1})^{n})$ denote the unique solution to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) (resp. for ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))). Then the angle $\angle(x_{i}(t),x_{j}(t))$ is the same for all distinct $i,j\in[n]$ :

|  | $$ \angle(x_{i}(t),x_{j}(t))=\theta_{\beta}(t) $$ |  |
| --- | --- | --- |

for $t\geqslant 0$ and some $\theta_{\beta}\in C^{0}(\mathbb{R}_{\geq 0};\mathbb{T})$ . Furthermore, for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), $\gamma_{\beta}(t):=\cos(\theta_{\beta}(t))$ satisfies

<table><tbody><tr><td rowspan="1"><span>(6.9)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><msub><mover><mi>γ</mi> <mo>˙</mo></mover> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mstyle><mfrac><mrow><mn>2</mn> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msup> <mo></mo><mrow><mo>(</mo><mrow><mn>1</mn> <mo>−</mo> <mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow> <mo></mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>(</mo><mrow><mi>n</mi> <mo>−</mo> <mn>1</mn></mrow><mo>)</mo></mrow> <mo></mo><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mn>1</mn></mrow><mo>)</mo></mrow></mrow> <mrow><msup><mi>e</mi> <mi>β</mi></msup> <mo>+</mo> <mrow><mrow><mo>(</mo><mrow><mi>n</mi> <mo>−</mo> <mn>1</mn></mrow><mo>)</mo></mrow> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msup></mrow></mrow></mfrac></mstyle></mrow></mtd> <mtd><mrow><mrow><mtext>for </mtext><mo></mo><mi>t</mi></mrow> <mo>≥</mo> <mn>0</mn></mrow></mtd></mtr> <mtr><mtd><mrow><mrow><mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>)</mo></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow><mo>,</mo></mrow></mtd></mtr></mtable></mrow><annotation>\displaystyle\begin{dcases}\dot{\gamma}_{\beta}(t)=\frac{2e^{\beta\gamma_{\beta}(t)}(1-\gamma_{\beta}(t))((n-1)\gamma_{\beta}(t)+1)}{e^{\beta}+(n-1)e^{\beta\gamma_{\beta}(t)}}&amp;\text{ for }t\geq 0\\ \gamma_{\beta}(0)=0\,,\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

and for ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), we have

<table><tbody><tr><td rowspan="1"><span>(6.10)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><msub><mover><mi>γ</mi> <mo>˙</mo></mover> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mstyle><mfrac><mn>2</mn> <mi>n</mi></mfrac></mstyle> <mo></mo><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></msup> <mo></mo><mrow><mo>(</mo><mrow><mn>1</mn> <mo>−</mo> <mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow> <mo></mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>(</mo><mrow><mi>n</mi> <mo>−</mo> <mn>1</mn></mrow><mo>)</mo></mrow> <mo></mo><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>+</mo> <mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow></mtd> <mtd><mrow><mrow><mtext>for </mtext><mo></mo><mi>t</mi></mrow> <mo>≥</mo> <mn>0</mn></mrow></mtd></mtr> <mtr><mtd><mrow><mrow><mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>)</mo></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow><mo>.</mo></mrow></mtd></mtr></mtable></mrow><annotation>\displaystyle\begin{dcases}\dot{\gamma}_{\beta}(t)=\frac{2}{n}e^{\beta\gamma_{\beta}(t)}(1-\gamma_{\beta}(t))((n-1)\gamma_{\beta}(t)+1)&amp;\text{ for }t\geq 0\\ \gamma_{\beta}(0)=0\,.\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

Here and henceforth, $\mathbb{T}=\mathbb{R}/2\pi\mathbb{Z}$ denotes the one-dimensional torus. We provide a brief proof of [Theorem ˜ 6.8](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem8 "Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") just below. The following result then shows that when $d\gg n$ , $t\mapsto\gamma_{\beta}(t)$ is a valid approximation for $t\mapsto\langle x_{i}(t),x_{j}(t)\rangle$ for any distinct $i,j\in[n]$ .

###### Theorem 6.9.

Fix $\beta\geqslant 0$ and $n\geq 2$ . Then there exists some $d^{*}(n,\beta)\geq n$ such that for all $d\geq d^{*}(n,\beta)$ , the following holds. Consider a sequence $(x_{i}(0))_{i\in[n]}$ of $n$ i.i.d. uniformly distributed points on $\mathbb{S}^{d-1}$ , and let $(x_{i}(\cdot))_{i\in[n]}\in C^{0}(\mathbb{R}_{\geq 0};(\mathbb{S}^{d-1})^{n})$ denote the unique solution to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). Then there exist $C=C(n,\beta)>0$ and $\lambda=\lambda(n,\beta)>0$ , such that with probability at least $1-2n^{2}d^{-1/64}$ ,

<table><tbody><tr><td rowspan="1"><span>(6.11)</span></td><td></td><td><math><semantics><mrow><mrow><mo>|</mo> <mrow><mrow><mo>⟨</mo> <mrow><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow> <mo>−</mo> <mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow> <mo>|</mo></mrow> <mo>⩽</mo> <mrow><mi>min</mi> <mo>⁡</mo> <mrow><mo>{</mo> <mrow><mrow><mn>2</mn> <mo>⋅</mo> <mi>c</mi></mrow> <mo></mo><msup><mrow><mo>(</mo><mi>β</mi><mo>)</mo></mrow> <mrow><mi>n</mi> <mo></mo><mi>t</mi></mrow></msup> <mo></mo><msqrt><mfrac><mrow><mi>log</mi> <mo>⁡</mo> <mi>d</mi></mrow> <mi>d</mi></mfrac></msqrt></mrow><mo>,</mo><mrow><mi>C</mi> <mo></mo><msup><mi>e</mi> <mrow><mo>−</mo> <mrow><mi>λ</mi> <mo></mo><mi>t</mi></mrow></mrow></msup></mrow> <mo>}</mo></mrow></mrow></mrow> <annotation>\Big{|}\langle x_{i}(t),x_{j}(t)\rangle-\gamma_{\beta}(t)\Big{|}\leqslant\min\left\{2\cdot c(\beta)^{nt}\sqrt{\frac{\log d}{d}},Ce^{-\lambda t}\right\}</annotation></semantics></math></td><td></td></tr></tbody></table>

holds for any $i\neq j$ and $t\geq 0$ , where $c(\beta)=e^{10\max\{1,\beta\}}$ , and $\gamma_{\beta}$ is the unique solution to ([6.9](https://arxiv.org/html/2312.10794v5#S6.E9 "Equation 6.9 ‣ Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")).

Since the proof is rather lengthy, we defer it to Appendix [D](https://arxiv.org/html/2312.10794v5#A4 "Appendix D Proof of Theorem 6.9 ‣ Appendix ‣ A mathematical perspective on Transformers"). It relies on combining the stability of the flow with respect to the initial data (entailed by the Lipschitz nature of the vector field) with concentration of measure. An analogous statement also holds for ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), and more details can be found in Remark  [D.1](https://arxiv.org/html/2312.10794v5#A4.Thmtheorem1 "Remark D.1. ‣ Step 3. Proof of (6.11) ‣ Appendix D Proof of Theorem 6.9 ‣ Appendix ‣ A mathematical perspective on Transformers"), whereas the explicit values of $C$ and $\lambda$ can be found in ([D.15](https://arxiv.org/html/2312.10794v5#A4.E15 "Equation D.15 ‣ Step 3. Proof of (6.11) ‣ Appendix D Proof of Theorem 6.9 ‣ Appendix ‣ A mathematical perspective on Transformers")). The upper bound in ([6.11](https://arxiv.org/html/2312.10794v5#S6.E11 "Equation 6.11 ‣ Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) is of interest in regimes where $d$ and/or $t$ are sufficiently large as the error in ([6.11](https://arxiv.org/html/2312.10794v5#S6.E11 "Equation 6.11 ‣ Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) is trivially bounded by $2$ .

###### Proof of.

We split the proof in two parts. We focus on proving the result for the dynamics ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), since the same arguments readily apply to the dynamics ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")).

##### Part 1. The angle θβ(t)\\theta\_{\\beta}(t)

We first show there exists $\theta\in C^{0}(\mathbb{R}_{\geq 0};\mathbb{T})$ such that $\theta(t)=\angle(x_{i}(t),x_{j}(t))$ for any distinct $(i,j)\in[n]^{2}$ and $t\geq 0$ . Since the initial tokens are orthogonal (and thus $d\geq n$ ), we may consider an orthonormal basis $(e_{1},\ldots,e_{d})$ of $\mathbb{R}^{d}$ such that $x_{i}(0)=e_{i}$ for $i\in[n]$ . Let $\pi:[d]\rightarrow[d]$ be a permutation. By decomposing any $x\in\mathbb{S}^{d-1}$ in this basis, we define $P_{\pi}:\mathbb{S}^{d-1}\rightarrow\mathbb{S}^{d-1}$ as

|  | $$ P_{\pi}\left(\sum_{i=1}^{n}a_{i}e_{i}\right)=\sum_{i=1}^{n}a_{i}e_{\pi(i)}. $$ |  |
| --- | --- | --- |

Setting $y_{i}(t)=P_{\pi}(x_{i}(t))$ for $i\in[n]$ , we see that $y_{i}(t)$ solves ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) with initial condition $y_{i}(0)=P_{\pi}(x_{i}(0))$ . But $(x_{\pi(1)}(t),\ldots,x_{\pi(n)}(t))$ is a solution of ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) by permutation equivariance, and it has the same initial condition since $P_{\pi}(x_{i}(0))=x_{\pi(i)}(0)$ . Consequently, we deduce that $P_{\pi}(x_{i}(t))=x_{\pi(i)}(t)$ for any $t\geq 0$ and any $i\in[d]$ . Hence

|  | $$ \langle x_{i}(t),x_{j}(t)\rangle=\langle P_{\pi}(x_{i}(t)),P_{\pi}(x_{j}(t))\rangle=\langle x_{\pi(i)}(t),x_{\pi(j)}(t)\rangle $$ |  |
| --- | --- | --- |

which concludes the proof.

##### Part 2. The curve γβ(t)\\gamma\_{\\beta}(t)

By virtue of the orthogonality assumption we have $\gamma_{\beta}(0)=\cos(\theta_{\beta}(0))=0$ . To prove that $\gamma_{\beta}(t)$ satisfies ([6.9](https://arxiv.org/html/2312.10794v5#S6.E9 "Equation 6.9 ‣ Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) for the case of ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), recall that

|  | $\displaystyle\bm{\mathsf{P}}^{\perp}_{x_{i}(t)}(x_{j}(t))=x_{j}(t)-\langle x_{i}(t),x_{j}(t)\rangle\,x_{i}(t).$ |  |
| --- | --- | --- |

Then for $k\neq i$ ,

|  | $\displaystyle\dot{\gamma}_{\beta}(t)$ | $\displaystyle=2\langle\dot{x}_{i}(t),x_{k}(t)\rangle$ |  |
| --- | --- | --- | --- |
|  |  | $\displaystyle=2\sum_{\begin{subarray}{c}j=1\end{subarray}}^{n}\left(\frac{e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}}{\sum_{\ell=1}^{n}e^{\beta\langle x_{i}(t),x_{\ell}(t)\rangle}}\right)\left(\langle x_{j}(t),x_{k}(t)\rangle-\langle x_{i}(t),x_{j}(t)\rangle\langle x_{i}(t),x_{k}(t)\rangle\right).$ |  |

Since the denominator in the above expression is equal to $(n-1)e^{\beta\gamma_{\beta}(t)}+e^{\beta}$ , we end up with

|  | $\displaystyle\dot{\gamma}_{\beta}(t)$ | $\displaystyle=\frac{2e^{\beta\gamma_{\beta}(t)}}{(n-1)e^{\beta\gamma_{\beta}(t)}+e^{\beta}}\sum_{\begin{subarray}{c}j=1\end{subarray}}^{n}\Big{(}\langle x_{j}(t),x_{k}(t)\rangle-\langle x_{i}(t),x_{j}(t)\rangle\langle x_{i}(t),x_{k}(t)\rangle\Big{)}$ |  |
| --- | --- | --- | --- |
|  |  | $\displaystyle=\frac{2e^{\beta\gamma_{\beta}(t)}}{(n-1)e^{\beta\gamma_{\beta}(t)}+e^{\beta}}(1-\gamma_{\beta}(t)^{2}+(n-2)(\gamma_{\beta}(t)-\gamma_{\beta}(t)^{2})),$ |  |

as desired. ∎

#### 6.3. Metastability and a phase transition

An interesting byproduct of [Theorem ˜ 6.8](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem8 "Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [Theorem ˜ 6.9](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem9 "Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") is the fact that they provide an accurate approximation of the exact *phase transition curve* delimiting the clustering and non-clustering regimes, in terms of $t$ and $\beta$ . To be more precise, given an initial sequence $(x_{i}(0))_{i\in[n]}\in(\mathbb{S}^{d-1})^{n}$ of random points distributed independently according to the uniform distribution on $\mathbb{S}^{d-1}$ , and for any fixed $0<\delta\ll 1$ , we define the phase transition curve as the boundary

|  | $$ \Gamma_{d,\delta}=\partial\left\{t,\beta\geq 0\colon t=\operatorname*{arg\,inf}_{s\geq 0}\Big{(}\mathbb{P}(\langle x_{1}(s),x_{2}(s)\rangle\geq 1-\delta)=1-2n^{2}d^{-\frac{1}{64}}\Big{)}\right\} $$ |  |
| --- | --- | --- |

where $(x_{i}(\cdot))_{i\in[n]}$ denotes the solution to the corresponding Cauchy problem for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). (Here the choice of the first two particles instead of a random distinct pair is justified due to permutation equivariance.) [Theorem ˜ 6.9](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem9 "Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") then gives the intuition that over compact subsets of $(\mathbb{R}_{\geq 0})^{2}$ , $\Gamma_{d,\delta}$ should be well-approximated by

<table><tbody><tr><td rowspan="1"><span>(6.12)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mi>Γ</mi> <mrow><mi>∞</mi><mo>,</mo><mi>δ</mi></mrow></msub> <mo>=</mo> <mrow><mo>{</mo> <mrow><mrow><mi>t</mi><mo>,</mo><mi>β</mi></mrow> <mo>≥</mo> <mn>0</mn></mrow><mo>:</mo><mrow><mrow><msub><mi>γ</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mn>1</mn> <mo>−</mo> <mi>δ</mi></mrow></mrow> <mo>}</mo></mrow></mrow><mo>.</mo></mrow><annotation>\Gamma_{\infty,\delta}=\Big{\{}t,\beta\geq 0\colon\gamma_{\beta}(t)=1-\delta\Big{\}}.</annotation></semantics></math></td><td></td></tr></tbody></table>

![Refer to caption](https://arxiv.org/html/x13.png)

(a) d = 2 d=2

This is clearly seen in Figure [3](https://arxiv.org/html/2312.10794v5#S6.F3 "Figure 3 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), along with the fact that the resolution of this approximation increases with $d\to+\infty$ .

Figure [3](https://arxiv.org/html/2312.10794v5#S6.F3 "Figure 3 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") appears to contain more information than what we may gather from Theorem  [6.3](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem3 "Theorem 6.3. ‣ 6.1. Clustering at an exponential rate when 𝑑≥𝑛 ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"), Theorem  [6.8](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem8 "Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and Theorem  [6.9](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem9 "Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"). In particular, for small $d$ , we see the appearance of a zone (white/light blue in Figure  [3](https://arxiv.org/html/2312.10794v5#S6.F3 "Figure 3 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers")) of parameters $(t,\beta)$ for which the probability of particles being clustered is positive, but not close to one. A careful inspection of this region reveals that points are grouped in a finite number of clusters; see Figure [4](https://arxiv.org/html/2312.10794v5#S6.F4 "Figure 4 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers"). The presence of such a zone indicates the emergence of a long-time *metastable* state where points are clustered into several groups but eventually relax to a single cluster in long-time. This two-time-scale phenomenon is illustrated in Figure  [4](https://arxiv.org/html/2312.10794v5#S6.F4 "Figure 4 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and prompts us to formulate the following question.

###### Problem 1.

Do the dynamics enter a transient metastable state, in the sense that for $\beta\gg 1$ , all particles stay in the vicinity of $m<n$ clusters for long periods of time, before they all collapse to the final cluster $\{x^{*}\}$ ?

There have been important steps towards a systematic theory of metastability for gradient flows, with applications to nonlinear parabolic equations—typically reaction-diffusion equations such as the Allen-Cahn or Cahn-Hilliard equations \[[OR07](https://arxiv.org/html/2312.10794v5#bib.bibx96), [KO02](https://arxiv.org/html/2312.10794v5#bib.bibx74)\] —. While these tools to not readily apply to the current setup, they form an important starting point to answer this question.

![Refer to caption](https://arxiv.org/html/x19.png)

Figure 4. We zoom in on the phase diagram (Figure 3 ) for the dynamics on the circle: d = 2 d=2. For β 4, 9 \\beta=4,9, we also display a trajectory of ( SA ) for a randomly drawn initial condition at times t 2.5 18 30 t=2.5,18,30. We see that the particles settle at clusters when \\beta=4 (bottom right) and \\beta=9 (top right), for a duration of time. This reflects our metastability claim for large \\beta

Finally, one may naturally ask whether the clustering and phase diagram conclusions persist when the parameter matrices $(Q,K,V)$ are significantly more general: some illustrations <sup>12</sup> <sup>12</sup> 12 See [github.com/borjanG/2023-transformers-rotf](https://github.com/borjanG/2023-transformers-rotf) for additional figures which indicate that this phenomenon appears to hold in even more generality. are given in Figure [5](https://arxiv.org/html/2312.10794v5#S6.F5 "Figure 5 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers").

###### Problem 2.

Can the conclusions of [Theorem ˜ 6.8](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem8 "Theorem 6.8. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") – [Theorem ˜ 6.9](https://arxiv.org/html/2312.10794v5#S6.Thmtheorem9 "Theorem 6.9. ‣ 6.2. More precise quantitative convergence ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") be generalized to the case of random matrices $(Q,K,V)$ ?

![Refer to caption](https://arxiv.org/html/x20.png)

(a) Q, K V Q,K,V in real Ginibre ensemble

## Part III Further questions

We conclude this manuscript by discussing several avenues of research that can lead to a finer understanding of the clustering phenomenon and generalizations of our results, and which, we believe, are of independent mathematical interest. Specifically,

- In Section [7](https://arxiv.org/html/2312.10794v5#S7 "7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers"), we zero in on the special case $d=2$ , where we make a link with the celebrated *Kuramoto model* when $\beta=0$ ;
- In Section [8](https://arxiv.org/html/2312.10794v5#S8 "8. BBGKY hierarchy ‣ Part III Further questions ‣ A mathematical perspective on Transformers"), we discuss an alternative approach for analyzing clustering, based on the so-called *BBGKY hierarchy* from statistical mechanics;
- In Section [9](https://arxiv.org/html/2312.10794v5#S9 "9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"), we foray beyond the case $Q^{\top}K=V=I_{d}$ in a few directions. We start with Section [9.1](https://arxiv.org/html/2312.10794v5#S9.SS1 "9.1. The repulsive case ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") by relating the case $V=-I_{d}$ to *optimal configurations* on the sphere. We then discuss existing results in the absence of layer normalization in Section [9.2](https://arxiv.org/html/2312.10794v5#S9.SS2 "9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"), which motivate the study of a related singular equation (Section [9.3](https://arxiv.org/html/2312.10794v5#S9.SS3 "9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) and a diffusive regularization (Section [9.4](https://arxiv.org/html/2312.10794v5#S9.SS4 "9.4. Diffusive regularization ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"));
- Finally, in Section [10](https://arxiv.org/html/2312.10794v5#S10 "10. Approximation, control, training ‣ Part III Further questions ‣ A mathematical perspective on Transformers"), we briefly discuss various ways focusing on the tuning of the parameter matrices themselves.

### 7\. Dynamics on the circle

We study the dynamics ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) and ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) in the special case $d=2$ , namely on the unit circle $\mathbb{S}^{1}\subset\mathbb{R}^{2}$ . This model, parametrized by angles and related to the celebrated Kuramoto model, is of independent interest and deserves a complete mathematical analysis.

#### 7.1. Angular equations

On the circle $\mathbb{S}^{1}$ , all particles $x_{i}(t)\in\mathbb{S}^{1}$ are of course completely characterized by the angle $\theta_{i}(t)\in\mathbb{T}$ : $x_{i}(t)=\cos(\theta_{i}(t))e_{1}+\sin(\theta_{i}(t))e_{2}$ where $e_{1}=(1,0)$ and $e_{2}=(0,1)\in\mathbb{R}^{2}$ . We focus on the dynamics ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) for simplicity. For any $i\in[n]$ and $t\geq 0$ , we may derive the equation satisfied by $\theta_{i}(t)$ from $\cos(\theta_{i}(t))=\langle x_{i}(t),e_{1}\rangle$ : differentiating in $t$ and plugging into ([USA](https://arxiv.org/html/2312.10794v5#S3.Ex10 "Equation USA ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")) we obtain

|  | $\displaystyle\dot{\theta}_{i}(t)=-\frac{n^{-1}}{\sin(\theta_{i}(t))}\left(\sum_{j=1}^{n}e^{\beta\langle x_{i}(t),x_{j}(t)\rangle}\Big{[}\langle x_{j}(t),e_{1}\rangle-\langle x_{i}(t),x_{j}(t)\rangle\langle x_{i}(t),e_{1}\rangle\Big{]}\right)$ |  |
| --- | --- | --- |

where we used the definition of the projection (if $\theta_{i}(t)=0$ for some $t$ , we differentiate the equality $\sin(\theta_{i}(t))=\langle x_{i}(t),e_{2}\rangle$ instead, which also leads to ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) in the end). Observing that

|  | $$ \langle x_{i}(t),x_{j}(t)\rangle=\cos(\theta_{i}(t)-\theta_{j}(t)), $$ |  |
| --- | --- | --- |

we find

|  | $\displaystyle\dot{\theta}_{i}(t)$ | $\displaystyle=-\frac{n^{-1}}{\sin(\theta_{i}(t))}\left(\sum_{j=1}^{n}e^{\beta\cos(\theta_{i}(t)-\theta_{j}(t))}\Big{[}\cos(\theta_{j}(t))-\cos(\theta_{i}(t)-\theta_{j}(t))\cos(\theta_{i}(t))\Big{]}\right).$ |  |
| --- | --- | --- | --- |

Using elementary trigonometry, we conclude that

<table><tbody><tr><td rowspan="1"><span>(7.1)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>θ</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mo>−</mo> <mrow><mfrac><mn>1</mn> <mi>n</mi></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mi>cos</mi> <mo>⁡</mo> <mrow><mo>(</mo><mrow><mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>θ</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></msup> <mo></mo><mrow><mi>sin</mi> <mo>⁡</mo> <mrow><mo>(</mo><mrow><mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>θ</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\dot{\theta}_{i}(t)=-\frac{1}{n}\sum_{j=1}^{n}e^{\beta\cos(\theta_{i}(t)-\theta_{j}(t))}\sin(\theta_{i}(t)-\theta_{j}(t)).</annotation></semantics></math></td><td></td></tr></tbody></table>

The case $\beta=0$ is exactly the Kuramoto model recalled in Section  [7.2](https://arxiv.org/html/2312.10794v5#S7.SS2 "7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers"). Suppose for the time being that $\beta>0$ . Defining the function $h_{\beta}:\mathbb{T}\to\mathbb{R}_{\geq 0}$ as

|  | $$ h_{\beta}(\theta)=e^{\beta\cos(\theta)}, $$ |  |
| --- | --- | --- |

we have effectively deduced that the empirical measure of the angles, $\nu(t)=\frac{1}{n}\sum_{j=1}^{n}\delta_{\theta_{j}(t)}$ , which is a measure on the torus $\mathbb{T}$ , is a solution to the continuity equation

|  | $$ \partial_{t}\nu(t)+\partial_{\theta}(\mathcal{X}[\nu(t)]\nu(t))=0,\hskip 28.45274pt\text{ on }\mathbb{R}_{\geq 0}\times\mathbb{T}, $$ |  |
| --- | --- | --- |

where

|  | $$ \mathcal{X}[\nu](\theta)=\frac{1}{\beta}\Big{(}h_{\beta}^{\prime}\ast\nu\Big{)}(\theta). $$ |  |
| --- | --- | --- |

When the particles $x_{i}(t)$ follow ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), one readily checks that the same continuity equation is satisfied but rather with the field

|  | $$ \mathcal{X}[\nu](\theta)=\frac{1}{\beta}\left(\frac{h_{\beta}^{\prime}\ast\nu}{h_{\beta}\ast\nu}\right)(\theta). $$ |  |
| --- | --- | --- |

#### 7.2. The Kuramoto model

As mentioned above, when $\beta=0$ , ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) is a particular case of the Kuramoto model \[[Kur75](https://arxiv.org/html/2312.10794v5#bib.bibx77)\]:

<table><tbody><tr><td rowspan="1"><span>(7.2)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>θ</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msub><mi>ω</mi> <mi>i</mi></msub> <mo>+</mo> <mrow><mfrac><mi>K</mi> <mi>n</mi></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><mi>sin</mi> <mo>⁡</mo> <mrow><mo>(</mo><mrow><mrow><msub><mi>θ</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\dot{\theta}_{i}(t)=\omega_{i}+\frac{K}{n}\sum_{j=1}^{n}\sin(\theta_{j}(t)-\theta_{i}(t)),</annotation></semantics></math></td><td></td></tr></tbody></table>

where $K>0$ is a prescribed coupling constant, and $\omega_{i}\in\mathbb{T}$ are the intrinsic natural frequencies of the oscillators $\theta_{i}(t)$ . It is known that for sufficiently small coupling strength $K$ , the oscillators $\theta_{i}(t)$ in the Kuramoto model ([7.2](https://arxiv.org/html/2312.10794v5#S7.E2 "Equation 7.2 ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) do not synchronize in long-time. It is also known that when $K$ exceeds some critical threshold value, a phase transition occurs, leading to the synchronization of a fraction of the oscillators. If $K$ is chosen very large, there is total synchronization of the oscillators in long-time. For more on the mathematical aspects of the Kuramoto model, we refer the reader to the review papers \[[Str00](https://arxiv.org/html/2312.10794v5#bib.bibx107), [ABV <sup><span>+</span></sup> 05](https://arxiv.org/html/2312.10794v5#bib.bibx2), [HKPZ16](https://arxiv.org/html/2312.10794v5#bib.bibx64)\] (see also \[[CCH <sup><span>+</span></sup> 14](https://arxiv.org/html/2312.10794v5#bib.bibx19), [Chi15](https://arxiv.org/html/2312.10794v5#bib.bibx22), [FGVG16](https://arxiv.org/html/2312.10794v5#bib.bibx51), [DFGV18](https://arxiv.org/html/2312.10794v5#bib.bibx40), [HR20](https://arxiv.org/html/2312.10794v5#bib.bibx67), [TSS20](https://arxiv.org/html/2312.10794v5#bib.bibx115), [ABK <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx1), [MP22](https://arxiv.org/html/2312.10794v5#bib.bibx90)\] for a non-exhaustive list of other recent mathematical results on the subject).

When all the frequencies $\omega_{i}$ are equal to some given frequency, $\omega\in\mathbb{R}$ say, after a change of variable of the form $\theta_{i}(t)\leftarrow\theta_{i}(t)-\omega t$ , the dynamics in ([7.2](https://arxiv.org/html/2312.10794v5#S7.E2 "Equation 7.2 ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) become the gradient flow

|  | $$ \dot{\theta}(t)=n\nabla\mathsf{F}(\theta) $$ |  |
| --- | --- | --- |

where the energy $\mathsf{F}:\mathbb{T}^{n}\to\mathbb{R}_{\geq 0}$ reads

<table><tbody><tr><td rowspan="1"><span>(7.3)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><mi>𝖥</mi> <mo></mo><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mi>K</mi> <mrow><mn>2</mn> <mo></mo><msup><mi>n</mi> <mn>2</mn></msup></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>i</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><mi>cos</mi> <mo>⁡</mo> <mrow><mo>(</mo><mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo>−</mo> <msub><mi>θ</mi> <mi>j</mi></msub></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo>.</mo></mrow><annotation>\mathsf{F}(\theta)=\frac{K}{2n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}\cos(\theta_{i}-\theta_{j}).</annotation></semantics></math></td><td></td></tr></tbody></table>

The oscillators can be viewed as attempting to maximize this energy. The energy $\mathsf{F}$ is maximized when all the oscillators are synchronized, that is, $\theta_{i}=\theta^{*}$ for some $\theta^{*}\in\mathbb{T}$ and for all $i\in[n]$ . As the dynamics follow a gradient system, the equilibrium states are the critical points of the energy, namely those satisfying $\nabla\mathsf{F}(\theta)=0$ . The local maxima of $\mathsf{F}$ correspond to equilibrium states $\theta$ that are physically achievable, since small perturbations thereof return the system back to $\theta$ .

Some authors consider a variant of the Kuramoto model where the oscillators are interacting according to the edges of a graph. In other words, the coefficients $A_{ij}$ of the graph’s adjacency matrix are inserted in the sum in ([7.3](https://arxiv.org/html/2312.10794v5#S7.E3 "Equation 7.3 ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) as weights, and the dynamics are then the corresponding gradient flow. A recent line of work culminating with \[[ABK <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx1)\] has established that synchronization occurs with high probability for Erdős–Rényi graphs with parameter $p$ , for every $p$ right above the connectivity threshold.

Coming back to our dynamics ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")), we notice that it can also be written as a gradient flow on $\mathbb{T}^{n}$ :

|  | $$ \dot{\theta}(t)=n\nabla\mathsf{E}_{\beta}(\theta(t)), $$ |  |
| --- | --- | --- |

for the interaction energy $\mathsf{E}_{\beta}:\mathbb{T}^{n}\to\mathbb{R}_{\geq 0}$ defined as

<table><tbody><tr><td rowspan="1"><span>(7.4)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>𝖤</mi> <mi>β</mi></msub> <mo></mo><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><mn>2</mn> <mo></mo><mi>β</mi> <mo></mo><msup><mi>n</mi> <mn>2</mn></msup></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>i</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mi>cos</mi> <mo>⁡</mo> <mrow><mo>(</mo><mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo>−</mo> <msub><mi>θ</mi> <mi>j</mi></msub></mrow><mo>)</mo></mrow></mrow></mrow></msup></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\mathsf{E}_{\beta}(\theta)=\frac{1}{2\beta n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}e^{\beta\cos(\theta_{i}-\theta_{j})},</annotation></semantics></math></td><td></td></tr></tbody></table>

which is maximized when $\theta_{i}=\theta^{*}$ for some $\theta^{*}\in\mathbb{T}$ and for all $i\in[n]$ . In the spirit of \[[LXB19](https://arxiv.org/html/2312.10794v5#bib.bibx88)\], we suggest the following open problem—we recall that a critical point is called a *strict saddle point* of $\mathsf{E}_{\beta}$ if the Hessian of $\mathsf{E}_{\beta}$ at these points has at least one positive eigenvalue.

###### Problem 3.

With the exception of the global maxima, are all critical points of $\mathsf{E}_{\beta}$ strict saddle points?

The proofs of Theorems [4.3](https://arxiv.org/html/2312.10794v5#S4.Thmtheorem3 "Theorem 4.3. ‣ 4.2. The case 𝛽≪1 ‣ 4. A single cluster for small 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") and [5.1](https://arxiv.org/html/2312.10794v5#S5.Thmtheorem1 "Theorem 5.1. ‣ 5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers") already yield a positive answer to Problem 5 in the regimes $\beta\leq 1$ or $\beta\geq\frac{n^{2}}{\pi^{2}}$ . The complementary regime remains open. By classical arguments, recalled in Appendix [A](https://arxiv.org/html/2312.10794v5#A1 "Appendix A Proof of Theorem 4.1 ‣ Appendix ‣ A mathematical perspective on Transformers"), a positive answer to Problem [3](https://arxiv.org/html/2312.10794v5#Thmopen3 "Problem 3. ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers") would imply that for all initial conditions except a set of measure zero, all $\theta_{i}(t)$ converge under the dynamics ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) to a common limit as $t\rightarrow+\infty$ .

Extensions of the Kuramoto model of the form

<table><tbody><tr><td rowspan="1"><span>(7.5)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>θ</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msub><mi>ω</mi> <mi>i</mi></msub> <mo>+</mo> <mrow><mfrac><mi>K</mi> <mi>n</mi></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><mi>h</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><msub><mi>θ</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>θ</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\dot{\theta}_{i}(t)=\omega_{i}+\frac{K}{n}\sum_{j=1}^{n}h(\theta_{j}(t)-\theta_{i}(t)),</annotation></semantics></math></td><td></td></tr></tbody></table>

for a general non-linearity $h:\mathbb{T}\to\mathbb{R}$ , which contains both ([7.2](https://arxiv.org/html/2312.10794v5#S7.E2 "Equation 7.2 ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) and our model ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) as particular cases, have already been studied in the physics literature. For instance, we refer the reader to \[[Dai92](https://arxiv.org/html/2312.10794v5#bib.bibx36)\] (see also \[[ABV <sup><span>+</span></sup> 05](https://arxiv.org/html/2312.10794v5#bib.bibx2), page 158\]), where many heuristics are proposed to address the behavior of solutions to these dynamics. We are not aware of mathematical results for ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) besides Theorem [5.1](https://arxiv.org/html/2312.10794v5#S5.Thmtheorem1 "Theorem 5.1. ‣ 5. A single cluster for large 𝛽 ‣ Part II Clustering ‣ A mathematical perspective on Transformers"). We nevertheless have some hope that handling the dynamics ([7.1](https://arxiv.org/html/2312.10794v5#S7.E1 "Equation 7.1 ‣ 7.1. Angular equations ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) is easier than dealing with ([7.5](https://arxiv.org/html/2312.10794v5#S7.E5 "Equation 7.5 ‣ 7.2. The Kuramoto model ‣ 7. Dynamics on the circle ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) for a general $h$ ; for instance, we have

|  | $$ h_{\beta}(\theta)=e^{\beta\cos(\theta)}=\sum_{k\in\mathbb{Z}}I_{k}(\beta)e^{\mathrm{i}k\theta} $$ |  |
| --- | --- | --- |

where $I_{k}(\beta)$ are the modified Bessel function of the first kind, whose properties have been extensively studied.

### 8\. BBGKY hierarchy

For the sake of simplicity, we again focus on the dynamics on the circle $\mathbb{S}^{1}$ , where recall that all particles are parametrized by angles (which we also refer to as particles). To carve out an even more complete understanding of the clustering phenomenon, it is natural to consider initial particles sampled i.i.d. from the uniform distribution on $\mathbb{S}^{1}$ and to study the time-evolution of the $r$ -particle distribution $\rho^{(r)}_{n}(t,\theta_{1},\ldots,\theta_{r})$ , defined as the joint law of the particles $\theta_{1}(t),\ldots,\theta_{r}(t)$ . Otherwise put, it is the $r$ -point marginal of the joint distribution $\rho^{(n)}(t,\cdot)\in\mathcal{P}(\mathbb{T}^{n})$ of all $n$ particles. Note that because of rotational invariance, $\rho^{(1)}(t,\cdot)$ is just the uniform distribution equal to $\frac{1}{2\pi}$ for all $t\geq 0$ . For $r=2$ , again by rotational invariance, there exists some $\psi(t,\cdot):\mathbb{T}\rightarrow\mathbb{R}_{\geq 0}$ such that

|  | $$ \rho^{(2)}(t,\theta_{1},\theta_{2})=\frac{1}{2\pi}\psi(t,\theta_{2}-\theta_{1}). $$ |  |
| --- | --- | --- |

Proving the clustering/synchronization of all $\theta_{i}(t)$ in long-time amounts to proving that $\psi(t,\cdot)$ converges to a Dirac mass centered at $0$ as $t\rightarrow+\infty$ . Using the fact that $\rho^{(n)}(t,\cdot)$ solves the Liouville equation, by following the method used to derive the BBGKY <sup>13</sup> <sup>13</sup> 13 Bogoliubov–Born–Green–Kirkwood–Yvon. hierarchy \[[GSRT13](https://arxiv.org/html/2312.10794v5#bib.bibx59), [Gol16](https://arxiv.org/html/2312.10794v5#bib.bibx58)\], it is possible to show that $\psi(t,\cdot)$ satisfies

<table><tbody><tr><td rowspan="1"><span>(8.1)</span></td><td></td><td><math><semantics><mrow><mo>{</mo> <mtable><mtr><mtd><mrow><mrow><mrow><msub><mo>∂</mo> <mi>t</mi></msub> <mrow><mi>ψ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow></mrow> <mo>+</mo> <mrow><msub><mo>∂</mo> <mi>x</mi></msub> <mrow><mo>(</mo><mrow><mi>v</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow> <mo></mo><mi>ψ</mi> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow></mrow> <mo>=</mo> <mn>0</mn></mrow></mtd> <mtd><mrow><mrow><mtext>in </mtext><mo></mo><msub><mi>ℝ</mi> <mrow><mo>≥</mo> <mn>0</mn></mrow></msub></mrow> <mo>×</mo> <mi>𝕋</mi></mrow></mtd></mtr> <mtr><mtd><mrow><mrow><mi>ψ</mi> <mo></mo><mrow><mo>(</mo><mn>0</mn><mo>,</mo><mi>x</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <msup><mrow><mo>(</mo><mrow><mn>2</mn> <mo></mo><mi>π</mi></mrow><mo>)</mo></mrow> <mrow><mo>−</mo> <mn>1</mn></mrow></msup></mrow></mtd> <mtd><mrow><mrow><mtext>in </mtext><mo></mo><mi>𝕋</mi></mrow><mo>,</mo></mrow></mtd></mtr></mtable></mrow><annotation>\begin{dcases}\partial_{t}\psi(t,x)+\partial_{x}(v(t,x)\psi(t,x))=0&amp;\text{ in }\mathbb{R}_{\geq 0}\times\mathbb{T}\\ \psi(0,x)=(2\pi)^{-1}&amp;\text{ in }\mathbb{T},\end{dcases}</annotation></semantics></math></td><td></td></tr></tbody></table>

where

|  | $$ v(t,x)=\frac{2}{\beta n}h_{\beta}^{\prime}(x)-\frac{2(n-2)}{\beta n}g(t,x), $$ |  |
| --- | --- | --- |

and

|  | $$ g(t,x)=\mathbb{E}\Big{[}-h_{\beta}^{\prime}(\theta_{3}(t))\,\Big{\|}\,\theta_{1}(t)=0,\ \theta_{2}(t)=x\Big{]}. $$ |  |
| --- | --- | --- | --- |

Note that the equation ([8.1](https://arxiv.org/html/2312.10794v5#S8.E1 "Equation 8.1 ‣ 8. BBGKY hierarchy ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) is not closed since $g(t,x)$ depends on the $3$ -point correlation function. This is typical in the BBGKY hierarchy, whereupon physical theory and experimental evidence is typically used to devise an ansatz for closing the system. For instance, the Boltzmann equation is derived from the BBGKY hierarchy by assuming the *molecular chaos hypothesis (Stosszahlansatz)* at the level of $r=2$ . We suggest to close ([8.1](https://arxiv.org/html/2312.10794v5#S8.E1 "Equation 8.1 ‣ 8. BBGKY hierarchy ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) in a way that reflects the formation of clusters:

###### Problem 4.

Devise a realistic ansatz for $g(t,x)$ which allows to close equation ([8.1](https://arxiv.org/html/2312.10794v5#S8.E1 "Equation 8.1 ‣ 8. BBGKY hierarchy ‣ Part III Further questions ‣ A mathematical perspective on Transformers")), and allows to prove the convergence of $\psi(t,\cdot)$ to a Dirac mass centered at $0$ as $t\to+\infty$ .

The derivation of a BBGKY hierarchy when $d>2$ , as well as for ([SA](https://arxiv.org/html/2312.10794v5#S2.Ex2 "Equation SA ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers")), are also problems which we believe merit further investigation.

### 9\. General matrices

Figure [5](https://arxiv.org/html/2312.10794v5#S6.F5 "Figure 5 ‣ 6.3. Metastability and a phase transition ‣ 6. The high-dimensional case ‣ Part II Clustering ‣ A mathematical perspective on Transformers") hints at the likelihood of the clustering phenomenon being significantly more general than just the case $Q=K=V=I_{d}$ . However, extending our proofs to more general parameter matrices does not appear to be straightforward and is an open problem. Here we discuss some particular cases (without excluding other approaches).

#### 9.1. The repulsive case

As seen from Lemma [3.7](https://arxiv.org/html/2312.10794v5#S3.Thmtheorem7 "Lemma 3.7. ‣ 3.3. A Wasserstein gradient flow proxy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers"), in the repulsive case $V=-I_{d}$ the interaction energy $\mathsf{E}_{\beta}$ decreases along trajectories. Recall that the unique global minimum of $\mathsf{E}_{\beta}$ over $\mathcal{P}(\mathbb{S}^{d-1})$ is the uniform distribution (Proposition [3.4](https://arxiv.org/html/2312.10794v5#S3.Thmtheorem4 "Proposition 3.4. ‣ 3.2. The interaction energy ‣ 3. Measure to measure flow map ‣ Part I Modeling ‣ A mathematical perspective on Transformers")). In contrast, we explain in this section that many different configurations of $n$ points may yield global minima for $\mathsf{E}_{\beta}$ when minimized over empirical measures with $n$ atoms.

We thus focus on minimizing $\mathsf{E}_{\beta}$ over the set $\mathcal{P}_{n}(\mathbb{S}^{d-1})$ of empirical measures, namely sums of $n$ Dirac masses. Rewriting $\mathsf{E}_{\beta}$ as

|  | $$ \mathsf{E}_{\beta}[\mu]=\frac{e^{\beta}}{2\beta}\iint e^{-\frac{\beta}{2}\\|x-x^{\prime}\\|^{2}}\,\mathrm{d}\mu(x)\,\mathrm{d}\mu(x^{\prime}), $$ |  |
| --- | --- | --- | --- | --- |

it turns out that minimizing $\mathsf{E}_{\beta}$ over $\mathcal{P}_{n}(\mathbb{S}^{d-1})$ is precisely the problem of finding *optimal configurations of points* on $\mathbb{S}^{d-1}$ , which has direct links to the sphere packing problem \[[CK07](https://arxiv.org/html/2312.10794v5#bib.bibx24), [CKM <sup><span>+</span></sup> 22](https://arxiv.org/html/2312.10794v5#bib.bibx25)\] and coding theory \[[DGS91](https://arxiv.org/html/2312.10794v5#bib.bibx42)\]. For $\mu\in\mathcal{P}_{n}(\mathbb{S}^{d-1})$ , we can equivalently rewrite $\mathsf{E}_{\beta}$ in terms of the set of support points $\mathscr{C}\subset\mathbb{S}^{d-1}$ , $\#\mathscr{C}=n$ :

|  | $$ \mathsf{E}_{\beta}[\mu]=\mathsf{H}_{\beta}[\mathscr{C}]=\frac{e^{\beta}}{2n^{2}\beta}\sum_{\begin{subarray}{c}x,x^{\prime}\in\mathscr{C}\end{subarray}}e^{-\frac{\beta}{2}\\|x-x^{\prime}\\|^{2}}. $$ |  |
| --- | --- | --- | --- | --- |

In \[[CK07](https://arxiv.org/html/2312.10794v5#bib.bibx24)\], Cohn and Kumar characterize the global minima $\mathscr{C}$ of $\mathsf{H}_{\beta}$ . To state their result, we need the following definition.

###### Definition 9.1.

Let $n\geq 2$ . A set of points $\mathscr{C}=\{x_{1},\ldots,x_{n}\}\subset\mathbb{S}^{d-1}$ is called a *spherical $t$ -design* if

|  | $$ \int p(x)\,\mathrm{d}\sigma_{d}(x)=\frac{1}{n}\sum_{i=1}^{n}p(x_{i}) $$ |  |
| --- | --- | --- |

for all polynomials $p$ of $d$ variables, of total degree at most $t$ . The set of points $\mathscr{C}$ is called a *sharp configuration* if there are $m$ distinct inner products between pairwise distinct points in $\mathscr{C}$ , for some $m>1$ , and if it is a spherical $(2m-1)$ -design.

The following result is a special case of \[[CK07](https://arxiv.org/html/2312.10794v5#bib.bibx24), Theorem 1.2\].

###### Theorem 9.2 (\[\]).

Let $n\geq 2$ . Any global minimum of $\mathsf{H}_{\beta}$ among $\mathscr{C}\subset\mathbb{S}^{d-1}$ , $\#\mathscr{C}=n$ is either a sharp configuration, or the vertices of a $600$ -cell <sup>14</sup> <sup>14</sup> 14 A $600$ \-cell is a particular $4$ \-dimensional convex polytope with $n=120$ vertices..

The set of sharp configurations is not known for all regimes of $n,d$ or $m$ (the largest $m$ such that the configuration is a spherical $m$ -design). A list of known examples is provided in \[[CK07](https://arxiv.org/html/2312.10794v5#bib.bibx24), Table 1\]: it consists of vertices of full-dimensional polytopes (specifically, regular polytopes whose faces are simplices), or particular derivations of the $E_{8}$ root lattice in $\mathbb{R}^{8}$ and the Leech lattice in $\mathbb{R}^{24}$ . We refer the reader to \[[CK07](https://arxiv.org/html/2312.10794v5#bib.bibx24)\] and the illustrative experimental paper \[[BBC <sup><span>+</span></sup> 09](https://arxiv.org/html/2312.10794v5#bib.bibx11)\] for further detail. A complete picture of the long-time behavior of Transformers in the repulsive case remains open.

#### 9.2. Pure self-attention

An alternative avenue for conducting such an analysis which has shown to be particularly fruitful consists in removing the projector $\bm{\mathsf{P}}^{\perp}_{x}$ , leading to

<table><tbody><tr><td rowspan="1"><span>(9.1)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>x</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>V</mi> <mo></mo><msub><mi>x</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></mrow></mrow> <annotation>\dot{x}_{i}(t)=\frac{1}{Z_{\beta,i}(t)}\sum_{j=1}^{n}e^{\beta\langle Qx_{i}(t),Kx_{j}(t)\rangle}Vx_{j}(t)</annotation></semantics></math></td><td></td></tr></tbody></table>

for all $i\in[n]$ and $t\in\mathbb{R}_{\geq 0}$ . In fact, in \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\] we analyze precisely these dynamics, and show different clustering results depending on the spectral properties of the matrix $V$ . We briefly summarize our findings in what follows.

##### 9.2.1. A review of \[\]

For most choices of value matrices $V$ , without rescaling time, most particles diverge to $\pm\infty$ and no particular pattern emerges. To make a very rough analogy, ([9.1](https://arxiv.org/html/2312.10794v5#S9.E1 "Equation 9.1 ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) "looks like" $\dot{x}_{i}(t)=Vx_{i}(t)$ (which amounts to having $P_{ij}(t)=\delta_{ij}$ instead of ([2.5](https://arxiv.org/html/2312.10794v5#S2.E5 "Equation 2.5 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))), whose solutions are given by $x_{i}(t)=e^{tV}x_{i}(0)$ . To discern the formation of clusters, we introduce the rescaling <sup>15</sup> <sup>15</sup> 15 The rescaling ([9.2](https://arxiv.org/html/2312.10794v5#S9.E2 "Equation 9.2 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) should be seen as a surrogate for layer normalization.

<table><tbody><tr><td rowspan="1"><span>(9.2)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><msup><mi>e</mi> <mrow><mo>−</mo> <mrow><mi>t</mi> <mo></mo><mi>V</mi></mrow></mrow></msup> <mo></mo><msub><mi>x</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>,</mo></mrow><annotation>z_{i}(t)=e^{-tV}x_{i}(t),</annotation></semantics></math></td><td></td></tr></tbody></table>

which are solutions to

<table><tbody><tr><td rowspan="1"><span>(9.3)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>z</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><msup><mi>e</mi> <mrow><mi>t</mi> <mo></mo><mi>V</mi></mrow></msup> <mo></mo><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><msup><mi>e</mi> <mrow><mi>t</mi> <mo></mo><mi>V</mi></mrow></msup> <mo></mo><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>V</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow></mrow> <annotation>\dot{z}_{i}(t)=\frac{1}{Z_{\beta,i}(t)}\sum_{j=1}^{n}e^{\beta\left\langle Qe^{tV}z_{i}(t),Ke^{tV}z_{j}(t)\right\rangle}V(z_{j}(t)-z_{i}(t))\hskip 28.45274pt</annotation></semantics></math></td><td></td></tr></tbody></table>

for $i\in[n]$ and $t\geq 0$ , where

|  | $$ Z_{\beta,i}(t)=\sum_{k=1}^{n}e^{\beta\langle Qe^{tV}z_{i}(t),Ke^{tV}z_{k}(t)\rangle}, $$ |  |
| --- | --- | --- |

whereas the initial condition remains the same, namely $x_{i}(0)=z_{i}(0)$ . It is crucial to notice that the coefficients $A_{ij}(t)$ (see ([2.5](https://arxiv.org/html/2312.10794v5#S2.E5 "Equation 2.5 ‣ 2.2. The interacting particle system ‣ 2. Interacting particle system ‣ Part I Modeling ‣ A mathematical perspective on Transformers"))) of the self-attention matrix for the rescaled particles $z_{i}(t)$ are the same as those for the original particles $x_{i}(t)$ . The weight $A_{ij}(t)$ indicates the strength of the attraction of $z_{i}(t)$ by $z_{j}(t)$ . In \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\] we show that the rescaled particles $z_{i}(t)$ cluster toward well-characterized geometric objects as $t\rightarrow+\infty$ for various choices of matrices $(Q,K,V)$ . Our results are summarized in Table [1](https://arxiv.org/html/2312.10794v5#S9.T1 "Table 1 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") below, whose first two lines are discussed thereafter.

| $V$ | $K$ and $Q$ | Limit geometry | Result in \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\] |
| --- | --- | --- | --- |
| $V=I_{d}$ | $Q^{\top}K\succ 0$ | vertices of convex polytope | Theorem 3.1 |
| $\lambda_{1}(V)>0$ , simple | $\langle Q\varphi_{1},K\varphi_{1}\rangle>0$ | union of $3$ parallel hyperplanes | Theorem 4.1 |
| $V$ paranormal | $Q^{\top}K\succ 0$ | polytope $\times$ subspaces | Theorem 5.1 |
| $V=-I_{d}$ | $Q^{\top}K=I_{d}$ | single cluster at origin <sup xmlns="http://www.w3.org/1999/xhtml"><span>∗</span></sup> | Theorem C.5 |

  

Table 1. Summary of the clustering results of \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\]. <sup>∗</sup> All results except for the case $V=-I_{d}$ hold for the time-scaled dynamics ([9.3](https://arxiv.org/html/2312.10794v5#S9.E3 "Equation 9.3 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")).

When $V=I_{d}$ , outside from exceptional situations, all particles cluster to vertices of some convex polytope. Indeed, since the velocity $\dot{z}_{i}(t)$ is a convex combination of the attractions $z_{j}(t)-z_{i}(t)$ , the convex hull $\mathcal{K}(t)$ of the $z_{i}(t)$ shrinks and thus converges to some convex polytope. The vertices of the latter attract all particles as $t\rightarrow+\infty$ . When the eigenvalue with largest real part of $V$ , denoted by $\lambda_{1}(V)$ , is simple and positive, the rescaled particles $z_{i}(t)$ cluster on hyperplanes which are parallel to the direct sum of the eigenspaces of the remaining eigenvalues. Roughly speaking, the coordinates of the points $z_{i}(t)$ along the eigenvector of $V$ corresponding to $\lambda_{1}(V)$ quickly dominate the matrix coefficients $P_{ij}(t)$ in ([9.3](https://arxiv.org/html/2312.10794v5#S9.E3 "Equation 9.3 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) due to the factors $e^{tV}z_{j}(t)$ . For more results and insights regarding clustering on $\mathbb{R}^{d}$ , we refer the reader to \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\]. We nonetheless leave the reader with the following general question:

###### Problem 5.

Is it possible to extend the clustering results of Table [1](https://arxiv.org/html/2312.10794v5#S9.T1 "Table 1 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") to other cases of $(Q,K,V)$ ? What are the resulting limit shapes?

#### 9.3. Singular dynamics

We mention another intriguing question, whose answer would allow for a transparent geometric understanding of clustering for ([9.3](https://arxiv.org/html/2312.10794v5#S9.E3 "Equation 9.3 ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")). Let $(Q,K,V)$ be given $d\times d$ matrices. For $\beta>0$ , we consider the system of coupled ODEs

<table><tbody><tr><td rowspan="1"><span>(9.4)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mover><mi>z</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><msub><mi>Z</mi> <mrow><mi>β</mi><mo>,</mo><mi>i</mi></mrow></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mfrac> <mo></mo><mrow><munderover><mo>∑</mo> <mrow><mi>j</mi> <mo>=</mo> <mn>1</mn></mrow> <mi>n</mi></munderover> <mrow><msup><mi>e</mi> <mrow><mi>β</mi> <mo></mo><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow></msup> <mo></mo><mi>V</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow><mo>,</mo></mrow><annotation>\dot{z}_{i}(t)=\frac{1}{Z_{\beta,i}(t)}\sum_{j=1}^{n}e^{\beta\langle Qz_{i}(t),Kz_{j}(t)\rangle}V(z_{j}(t)-z_{i}(t)),</annotation></semantics></math></td><td></td></tr></tbody></table>

where once again

|  | $$ Z_{\beta,i}(t)=\sum_{k=1}^{n}e^{\beta\langle Qz_{i}(t),Kz_{k}(t)\rangle}. $$ |  |
| --- | --- | --- |

For any $T>0$ , and any fixed initial condition $(z_{i}(0))_{i\in[n]}\in(\mathbb{R}^{d})^{n}$ , as $\beta\rightarrow+\infty$ , we expect that the solution to ([9.4](https://arxiv.org/html/2312.10794v5#S9.E4 "Equation 9.4 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) converges uniformly on $[0,T]$ to a solution of

<table><tbody><tr><td rowspan="1"><span>(9.5)</span></td><td></td><td><math><semantics><mrow><mrow><msub><mover><mi>z</mi> <mo>˙</mo></mover> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mfrac><mn>1</mn> <mrow><mo>|</mo> <mrow><msub><mi>C</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>|</mo></mrow></mfrac> <mo></mo><mrow><munder><mo>∑</mo> <mrow><mi>j</mi> <mo>∈</mo> <mrow><msub><mi>C</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow></munder> <mrow><mi>V</mi> <mo></mo><mrow><mo>(</mo><mrow><mrow><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>−</mo> <mrow><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mrow> <annotation>\dot{z}_{i}(t)=\frac{1}{|C_{i}(t)|}\sum_{j\in C_{i}(t)}V(z_{j}(t)-z_{i}(t))</annotation></semantics></math></td><td></td></tr></tbody></table>

where

<table><tbody><tr><td rowspan="1"><span>(9.6)</span></td><td></td><td><math><semantics><mrow><mrow><mrow><msub><mi>C</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>=</mo> <mrow><mo>{</mo> <mrow><mi>j</mi> <mo>∈</mo> <mrow><mo>[</mo><mi>n</mi><mo>]</mo></mrow></mrow><mo>:</mo><mrow><mrow><mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><msub><mi>z</mi> <mi>j</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow> <mo>≥</mo> <mrow><mo>⟨</mo> <mrow><mi>Q</mi> <mo></mo><msub><mi>z</mi> <mi>i</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>,</mo><mrow><mi>K</mi> <mo></mo><msub><mi>z</mi> <mi>k</mi></msub> <mo></mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow> <mo>⟩</mo></mrow></mrow> <mrow><mrow><mtext>for all </mtext><mo></mo><mi>k</mi></mrow> <mo>∈</mo> <mrow><mo>[</mo><mi>n</mi><mo>]</mo></mrow></mrow></mrow> <mo>}</mo></mrow></mrow><mo>.</mo></mrow><annotation>C_{i}(t)=\Big{\{}j\in[n]\colon\langle Qz_{i}(t),Kz_{j}(t)\rangle\geq\langle Qz_{i}(t),Kz_{k}(t)\rangle\quad\text{ for all }k\in[n]\Big{\}}.</annotation></semantics></math></td><td></td></tr></tbody></table>

However, defining a notion of solution to ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) is not straightforward, as illustrated by the following example.

###### Example 9.3.

Suppose $d=2$ , $n=3$ . Let $Q=K=V=I_{d}$ and $z_{1}(0)=(1,1)$ , $z_{2}(0)=(-1,1)$ , $z_{3}(0)=(0,0)$ . Consider the evolution of these particles through ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")). The points $z_{1}(t)$ and $z_{2}(t)$ do not move, because it is easily seen that $C_{i}(t)=\{i\}$ for $i\in\{1,2\}$ . On the other hand, the point $z_{3}(t)$ can be chosen to solve either of three equations: $\dot{z}_{3}(t)=z_{1}(t)-z_{3}(t)$ , or $\dot{z}_{3}(t)=z_{2}(t)-z_{3}(t)$ , or even $\dot{z}_{3}(t)=\frac{1}{2}(z_{1}(t)+z_{2}(t))-z_{3}(t)$ . In any of these cases, both ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) and ([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) remain satisfied for almost every $t\geq 0$ .

It is possible to prove the existence of solutions to ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) defined in the sense of Filippov <sup>16</sup> <sup>16</sup> 16 We thank Enrique Zuazua for this suggestion.: for this, we can either use a time-discretization of ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")), or use a convergence argument for solutions to ([9.4](https://arxiv.org/html/2312.10794v5#S9.E4 "Equation 9.4 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) as $\beta\rightarrow+\infty$ . Uniqueness however does not hold, as illustrated by Example  [9.3](https://arxiv.org/html/2312.10794v5#S9.Thmtheorem3 "Example 9.3. ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"). This naturally leads us to the following question:

###### Problem 6.

Is it possible to establish a selection principle (similar to viscosity or entropy solutions) for solutions to ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) which allows to restore uniqueness? In the affirmative, is it possible to revisit the clustering results of \[[GLPR24](https://arxiv.org/html/2312.10794v5#bib.bibx57)\] and Problem [5](https://arxiv.org/html/2312.10794v5#Thmopen5 "Problem 5. ‣ 9.2.1. A review of [GLPR24] ‣ 9.2. Pure self-attention ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers") in the setting of ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))?

#### 9.4. Diffusive regularization

We believe that ([9.5](https://arxiv.org/html/2312.10794v5#S9.E5 "Equation 9.5 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers"))–([9.6](https://arxiv.org/html/2312.10794v5#S9.E6 "Equation 9.6 ‣ 9.3. Singular dynamics ‣ 9. General matrices ‣ Part III Further questions ‣ A mathematical perspective on Transformers")) is also an original model for collective behavior. There are some similarities in spirit with methods arising in *consensus based optimization* (CBO for short), \[[PTTM17](https://arxiv.org/html/2312.10794v5#bib.bibx99), [CJLZ21](https://arxiv.org/html/2312.10794v5#bib.bibx23)\]. With CBO methods, one wishes to minimize a smooth and bounded, but otherwise arbitrary function $f:\mathbb{R}^{d}\to\mathbb{R}$ by making use of the Laplace method

|  | $$ \lim_{\beta\to+\infty}\left(-\frac{1}{\beta}\log\int_{\mathbb{R}^{d}}e^{-\beta f(x)}\,\mathrm{d}\rho(x)\right)=\inf_{x\in\text{supp}(\rho)}f(x), $$ |  |
| --- | --- | --- |

which holds for any fixed $\rho\in\mathcal{P}_{\text{ac}}(\mathbb{R}^{d})$ . This is accomplished by considering a McKean-Vlasov particle system of the form

|  | $$ \,\mathrm{d}x_{i}(t)=-\lambda(x_{i}(t)-v_{f})H^{\epsilon}(f(x_{i}(t))-f(v[\mu_{n}(t)]))\,\mathrm{d}t+\sqrt{2}\sigma\|x_{i}(t)-v[\mu_{n}(t)]\|\,\mathrm{d}W_{i}(t) $$ |  |
| --- | --- | --- | --- | --- |

for fixed $\beta>0$ , with drift parameter $\lambda>0$ and noise parameter $\sigma\geq 0$ ; $H^{\epsilon}$ is a particular smoothed Heaviside function, and $\mu_{n}(t)$ is the empirical measure of the particles. The point $v[\mu]\in\mathbb{R}^{d}$ is a weighted average of the particles:

|  | $$ v[\mu]=\frac{1}{Z_{\beta,\mu}}\int_{\mathbb{R}^{d}}e^{-\beta f(x)}x\,\mathrm{d}\mu(x) $$ |  |
| --- | --- | --- |

where $Z_{\beta,\mu}=\int_{\mathbb{R}^{d}}e^{-\beta f(x)}\,\mathrm{d}\mu(x)$ . Morally speaking, particles which are near a minimum of $f$ have a larger weight. The drift term is a gradient relaxation (for a quadratic potential) towards the current weighted average position of the batch of particles. The diffusion term is an exploration term whose strength is proportional to the distance of the particle from the current weighted average. Results of convergence to a global minimizer do exist, under various smallness assumptions on the initial distribution of the particles, and assumptions on the relative size of the coefficients. They rely on the analysis of the associated Fokker-Planck equation, see \[[CJLZ21](https://arxiv.org/html/2312.10794v5#bib.bibx23), [CD22](https://arxiv.org/html/2312.10794v5#bib.bibx20)\], and also \[[FHPS21](https://arxiv.org/html/2312.10794v5#bib.bibx52)\] for the analog on $\mathbb{S}^{d-1}$ . We point out that similarities are mainly in spirit—these results and analysis are inapplicable to our setting because there is no analog for $f(x)$ —. Nonetheless, they do raise the following interesting question:

###### Problem 7.

What can be said about the long-time limit of Transformers with a noise/diffusion term of strength $\sigma>0$ ?

The question is of interest for any of the Transformers models presented in what precedes.

### 10\. Approximation, control, training

Understanding the *expressivity*, namely the ability of a neural network to reproduce any map in a given class (by tuning its parameters), is essential. Two closely related notions reflect the expressivity of neural networks: *interpolation* —the property of exactly matching arbitrarily many input and target samples—and *(universal) approximation* —the property of approximating input-target functional relationships in an appropriate topology—. We refer the reader to \[[CLLS23](https://arxiv.org/html/2312.10794v5#bib.bibx26)\] for a primer on the relationship between these two notions in the contex of deep neural networks.

For discrete-time Transformers, universal approximation has been shown to hold in \[[YBR <sup><span>+</span></sup> 19](https://arxiv.org/html/2312.10794v5#bib.bibx127)\], making use of a variant of the architecture with translation parameters and letting the number of layers go to infinity; see also \[[ADTK23](https://arxiv.org/html/2312.10794v5#bib.bibx4), [JL23](https://arxiv.org/html/2312.10794v5#bib.bibx71)\] and the review \[[JLLW23](https://arxiv.org/html/2312.10794v5#bib.bibx72)\].

In the context of flow maps (from $\mathbb{R}^{d}$ to $\mathbb{R}^{d}$ ), it is now well understood that interpolation and approximation reflect the *controllability* properties of the system. The transfer of control theoretical techniques to the understanding of expressivity has borne fruit, both in terms of controllability results \[[AS22](https://arxiv.org/html/2312.10794v5#bib.bibx8), [CLT20](https://arxiv.org/html/2312.10794v5#bib.bibx28), [TG22](https://arxiv.org/html/2312.10794v5#bib.bibx113), [LLS22](https://arxiv.org/html/2312.10794v5#bib.bibx85), [RBZ23](https://arxiv.org/html/2312.10794v5#bib.bibx100), [VR23](https://arxiv.org/html/2312.10794v5#bib.bibx121), [CLLS23](https://arxiv.org/html/2312.10794v5#bib.bibx26)\] and optimal control insights \[[LCT18](https://arxiv.org/html/2312.10794v5#bib.bibx80), [EGPZ20](https://arxiv.org/html/2312.10794v5#bib.bibx48), [GZ22](https://arxiv.org/html/2312.10794v5#bib.bibx61), [EYG23](https://arxiv.org/html/2312.10794v5#bib.bibx49)\]. We refer the reader to \[[AL24](https://arxiv.org/html/2312.10794v5#bib.bibx7), [AG24](https://arxiv.org/html/2312.10794v5#bib.bibx5), [FdHP24](https://arxiv.org/html/2312.10794v5#bib.bibx50)\] for the first universal approximation results for Transformers, viewed as measure-to-measure maps, using control theoretic tools.

Besides approximation, understanding the training dynamics of Transformers is another major challenge which we haven’t covered herein. As it is impossible to reference all works on this burgeoning topic, we refer the interested reader to \[[TLTO23](https://arxiv.org/html/2312.10794v5#bib.bibx114), [ACDS23](https://arxiv.org/html/2312.10794v5#bib.bibx3), [DGTT24](https://arxiv.org/html/2312.10794v5#bib.bibx43)\] and references therein.

### Acknowledgments

We thank Pierre Ablin, Sébastien Bubeck, Gabriel Peyré, Matthew Rosenzweig, Sylvia Serfaty, Kimi Sun, and Rui Sun for discussions. We thank Nicolas Boumal for referring us to \[[MTG17](https://arxiv.org/html/2312.10794v5#bib.bibx92), [CRMB24](https://arxiv.org/html/2312.10794v5#bib.bibx32)\] and for clarifying comments.

B.G. acknowledges financial support from the French government managed by the National Agency for Research under the France 2030 program, with the reference ”ANR-23-PEIA-0004”. The work of Y.P. was supported in part by the MIT-IBM Watson AI Lab and by the National Science Foundation under Grant No CCF-2131115. P.R. was supported by NSF grants DMS-2022448, CCF-2106377, and a gift from Apple.

## Appendix

### References

[^1]: Pedro Abdalla, Afonso S Bandeira, Martin Kassabov, Victor Souza, Steven H Strogatz, and Alex Townsend.Expander graphs are globally synchronising.arXiv preprint arXiv:2210.12788, 2022.

[^2]: Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler.The Kuramoto model: A simple paradigm for synchronization phenomena.Reviews of Modern Physics, 77(1):137, 2005.

[^3]: Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra.Transformers learn to implement preconditioned gradient descent for in-context learning.Advances in Neural Information Processing Systems, 36, 2023.

[^4]: Silas Alberti, Niclas Dern, Laura Thesing, and Gitta Kutyniok.Sumformer: Universal Approximation for Efficient Transformers.In Topological, Algebraic and Geometric Learning Workshops 2023, pages 72–86. PMLR, 2023.

[^5]: Daniel Owusu Adu and Bahman Gharesifard.Approximate controllability of continuity equation of transformers.IEEE Control Systems Letters, 2024.

[^6]: Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré.Gradient flows: in metric spaces and in the space of probability measures.Springer Science & Business Media, 2005.

[^7]: Andrei Agrachev and Cyril Letrouit.Generic controllability of equivariant systems and applications to particle systems and neural networks.arXiv preprint arXiv:2404.08289, 2024.

[^8]: Andrei Agrachev and Andrey Sarychev.Control on the manifolds of mappings with a view to the deep learning.Journal of Dynamical and Control Systems, 28(4):989–1008, 2022.

[^9]: Andrew R Barron.Universal approximation bounds for superpositions of a sigmoidal function.IEEE Transactions on Information Theory, 39(3):930–945, 1993.

[^10]: Jean-David Benamou and Yann Brenier.A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem.Numerische Mathematik, 84(3):375–393, 2000.

[^11]: Brandon Ballinger, Grigoriy Blekherman, Henry Cohn, Noah Giansiracusa, Elizabeth Kelly, and Achill Schürmann.Experimental study of energy-minimizing point configurations on spheres.Experimental Mathematics, 18(3):257–283, 2009.

[^12]: Adrien Blanchet, José A Carrillo, and Nader Masmoudi.Infinite time aggregation for the critical Patlak-Keller-Segel model in $\mathbb{R}^{2}$ .Communications on Pure and Applied Mathematics, 61(10):1449–1481, 2008.

[^13]: Dario Benedetto, Emanuele Caglioti, and Umberto Montemagno.On the complete phase synchronization for the Kuramoto model in the mean-field limit.Communications in Mathematical Sciences, 13(7):1775–1786, 2015.

[^14]: Dmitriy Bilyk and Feng Dai.Geodesic distance Riesz energy on the sphere.Transactions of the American Mathematical Society, 372(5):3141–3166, 2019.

[^15]: Han Bao, Ryuichiro Hataya, and Ryo Karakida.Self-attention networks localize when qk-eigenspectrum concentrates.arXiv preprint arXiv:2402.02098, 2024.

[^16]: Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.Layer normalization.arXiv preprint arXiv:1607.06450, 2016.

[^17]: Emmanuel Boissard and Thibaut Le Gouic.On the mean speed of convergence of empirical and occupation measures in Wasserstein distance.In Annales de l’IHP Probabilités et statistiques, volume 50, pages 539–563, 2014.

[^18]: Andrea L Bertozzi, Thomas Laurent, and Jesús Rosado.$L^{p}$ theory for the multidimensional aggregation equation.Communications on Pure and Applied Mathematics, 64(1):45–83, 2011.

[^19]: José A Carrillo, Young-Pil Choi, Seung-Yeal Ha, Moon-Jin Kang, and Yongduck Kim.Contractivity of transport distances for the kinetic Kuramoto equation.Journal of Statistical Physics, 156(2):395–415, 2014.

[^20]: Louis-Pierre Chaintron and Antoine Diez.Propagation of chaos: a review of models, methods and applications. II. Applications.2022.

[^21]: J. A. Carrillo, M. DiFrancesco, A. Figalli, T. Laurent, and D. Slepčev.Global-in-time weak measure solutions and finite-time aggregation for nonlocal interaction equations.Duke Mathematical Journal, 156(2):229 – 271, 2011.

[^22]: Hayato Chiba.A proof of the Kuramoto conjecture for a bifurcation structure of the infinite-dimensional Kuramoto model.Ergodic Theory and Dynamical Systems, 35(3):762–834, 2015.

[^23]: José A Carrillo, Shi Jin, Lei Li, and Yuhua Zhu.A consensus-based global optimization method for high dimensional machine learning problems.ESAIM: Control, Optimisation and Calculus of Variations, 27:S5, 2021.

[^24]: Henry Cohn and Abhinav Kumar.Universally optimal distribution of points on spheres.Journal of the American Mathematical Society, 20(1):99–148, 2007.

[^25]: Henry Cohn, Abhinav Kumar, Stephen Miller, Danylo Radchenko, and Maryna Viazovska.Universal optimality of the $E_{8}$ and Leech lattices and interpolation formulas.Annals of Mathematics, 196(3):983–1082, 2022.

[^26]: Jingpu Cheng, Qianxiao Li, Ting Lin, and Zuowei Shen.Interpolation, approximation and controllability of deep neural networks.arXiv preprint arXiv:2309.06015, 2023.

[^27]: Marco Caponigro, Anna Chiara Lai, and Benedetto Piccoli.A nonlinear model of opinion formation on the sphere.Discrete & Continuous Dynamical Systems-A, 35(9):4241–4268, 2015.

[^28]: Christa Cuchiero, Martin Larsson, and Josef Teichmann.Deep neural networks, generic universal interpolation, and controlled ODEs.SIAM Journal on Mathematics of Data Science, 2(3):901–919, 2020.

[^29]: Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi, and Surya Ganguli.Geometric dynamics of signal propagation predict trainability of transformers.arXiv preprint arXiv:2403.02579, 2024.

[^30]: Sinho Chewi, Jonathan Niles-Weed, and Philippe Rigollet.Statistical optimal transport.arXiv preprint arXiv:2407.18163, 2024.

[^31]: Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.Neural ordinary differential equations.Advances in Neural Information Processing Systems, 31, 2018.

[^32]: Christopher Criscitiello, Quentin Rebjock, Andrew D. McRae, and Nicolas Boumal.Synchronization on circles and spheres with nonlinear interactions, 2024.

[^33]: Felipe Cucker and Steve Smale.Emergent behavior in flocks.IEEE Transactions on Automatic Control, 52(5):852–862, 2007.

[^34]: George Cybenko.Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989.

[^35]: Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang.The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020–12030, 2022.

[^36]: Hiroaki Daido.Order function and macroscopic mutual entrainment in uniformly coupled limit-cycle oscillators.Progress of Theoretical Physics, 88(6):1213–1218, 1992.

[^37]: Gbètondji JS Dovonon, Michael M Bronstein, and Matt J Kusner.Setting the record straight on transformer oversmoothing.arXiv preprint arXiv:2401.04301, 2024.

[^38]: Gwendoline De Bie, Gabriel Peyré, and Marco Cuturi.Stochastic deep networks.In International Conference on Machine Learning, pages 1556–1565. PMLR, 2019.

[^39]: Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas.Attention is not all you need: Pure attention loses rank doubly exponentially with depth.In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021.

[^40]: Helge Dietert, Bastien Fernandez, and David Gérard-Varet.Landau damping to partially locked states in the Kuramoto model.Communications on Pure and Applied Mathematics, 71(5):953–993, 2018.

[^41]: Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty.Redesigning the transformer architecture with insights from multi-particle dynamical systems.Advances in Neural Information Processing Systems, 34:5531–5544, 2021.

[^42]: Philippe Delsarte, Jean-Marie Goethals, and Johan Jacob Seidel.Spherical codes and designs.In Geometry and Combinatorics, pages 68–93. Elsevier, 1991.

[^43]: Puneesh Deora, Rouzbeh Ghaderi, Hossein Taheri, and Christos Thrampoulidis.On the optimization and generalization of multi-head attention.Transactions on Machine Learning Research, 2024.

[^44]: Roland L’vovich Dobrushin.Vlasov equations.Funktsional’nyi Analiz i ego Prilozheniya, 13(2):48–58, 1979.

[^45]: R. M. Dudley.The Speed of Mean Glivenko-Cantelli Convergence.The Annals of Mathematical Statistics, 40(1):40 – 50, 1969.

[^46]: Feng Dai and Yuan Xu.Approximation theory and harmonic analysis on spheres and balls.Springer, 2013.

[^47]: Weinan E.A proposal on machine learning via dynamical systems.Communications in Mathematics and Statistics, 1(5):1–11, 2017.

[^48]: Carlos Esteve, Borjan Geshkovski, Dario Pighin, and Enrique Zuazua.Large-time asymptotics in deep learning.arXiv preprint arXiv:2008.02491, 2020.

[^49]: Carlos Esteve-Yagüe and Borjan Geshkovski.Sparsity in long-time control of neural ODEs.Systems & Control Letters, 172:105452, 2023.

[^50]: Takashi Furuya, Maarten V de Hoop, and Gabriel Peyré.Transformers are universal in-context learners.arXiv preprint arXiv:2408.01367, 2024.

[^51]: Bastien Fernandez, David Gérard-Varet, and Giambattista Giacomin.Landau damping in the Kuramoto model.In Annales Henri Poincaré, volume 17, pages 1793–1823. Springer, 2016.

[^52]: Massimo Fornasier, Hui Huang, Lorenzo Pareschi, and Philippe Sünnen.Consensus-based optimization on the sphere: Convergence to global minimizers and machine learning.The Journal of Machine Learning Research, 22(1):10722–10776, 2021.

[^53]: Amic Frouvelle and Jian-Guo Liu.Long-time dynamics for a simple aggregation equation on the sphere.In Stochastic Dynamics Out of Equilibrium: Institut Henri Poincaré, Paris, France, 2017, pages 457–479. Springer, 2019.

[^54]: Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha.Rank diminishing in deep neural networks.Advances in Neural Information Processing Systems, 35:33054–33065, 2022.

[^55]: Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning.MIT Press, Cambridge, MA, 2016.

[^56]: Arnaud Guillin, Pierre Le Bris, and Pierre Monmarché.Uniform in time propagation of chaos for the 2D vortex model and other singular stochastic systems.Journal of the European Mathematical Society, 2024.

[^57]: Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet.The emergence of clusters in self-attention dynamics.Advances in Neural Information Processing Systems, 36, 2024.

[^58]: François Golse.On the dynamics of large particle systems in the mean field limit.Macroscopic and large scale phenomena: coarse graining, mean field limits and ergodicity, pages 1–144, 2016.

[^59]: Isabelle Gallagher, Laure Saint-Raymond, and Benjamin Texier.From Newton to Boltzmann: hard spheres and short-range potentials.European Mathematical Society Zürich, Switzerland, 2013.

[^60]: Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang.Contranorm: A contrastive learning perspective on oversmoothing and beyond.In The Eleventh International Conference on Learning Representations, 2023.

[^61]: Borjan Geshkovski and Enrique Zuazua.Turnpike in optimal control of PDEs, ResNets, and beyond.Acta Numerica, 31:135–263, 2022.

[^62]: Jiequn Han, Ruimeng Hu, and Jihao Long.A class of dimension-free metrics for the convergence of empirical measures.Stochastic Processes and their Applications, 164:242–287, 2023.

[^63]: Rainer Hegselmann and Ulrich Krause.Opinion dynamics and bounded confidence: models, analysis and simulation.Journal of Artifical Societies and Social Simulation (JASSS), 5(3), 2002.

[^64]: Seung-Yeal Ha, Dongnam Ko, Jinyeong Park, and Xiongtao Zhang.Collective synchronization of classical and quantum oscillators.EMS Surveys in Mathematical Sciences, 3(2):209–267, 2016.

[^65]: Seung-Yeal Ha, Dongnam Ko, and Sang Woo Ryoo.On the relaxation dynamics of Lohe oscillators on some Riemannian manifolds.Journal of Statistical Physics, 172:1427–1478, 2018.

[^66]: Eldad Haber and Lars Ruthotto.Stable architectures for deep neural networks.Inverse problems, 34(1), 2017.

[^67]: Seung-Yeal Ha and Seung-Yeon Ryoo.Asymptotic phase-locking dynamics and critical coupling strength for the Kuramoto model.Communications in Mathematical Physics, 377(2):811–857, 2020.

[^68]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Identity mappings in deep residual networks.In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.

[^69]: Amir Joudaki, Hadi Daneshmand, and Francis Bach.On the impact of activation and normalization in obtaining isometric embeddings at initialization.Advances in Neural Information Processing Systems, 36:39855–39875, 2023.

[^70]: Richard Jordan, David Kinderlehrer, and Felix Otto.The variational formulation of the Fokker–Planck equation.SIAM Journal on Mathematical Analysis, 29(1):1–17, 1998.

[^71]: Haotian Jiang and Qianxiao Li.Approximation theory of transformer networks for sequence modeling.arXiv preprint arXiv:2305.18475, 2023.

[^72]: Haotian Jiang, Qianxiao Li, Zhong Li, and Shida Wang.A brief survey on the approximation theory for sequence modelling.arXiv preprint arXiv:2302.13752, 2023.

[^73]: Pierre-Emmanuel Jabin and Sebastien Motsch.Clustering and asymptotic behavior in opinion formation.Journal of Differential Equations, 257(11):4165–4187, 2014.

[^74]: Robert V Kohn and Felix Otto.Upper bounds on coarsening rates.Communications in Mathematical Physics, 229(3):375–395, 2002.

[^75]: Ulrich Krause.A discrete nonlinear and non-autonomous model of consensus.In Communications in Difference Equations: Proceedings of the Fourth International Conference on Difference Equations, page 227. CRC Press, 2000.

[^76]: Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.Imagenet classification with deep convolutional neural networks.Advances in Neural Information Processing Systems, 25, 2012.

[^77]: Yoshiki Kuramoto.Self-entrainment of a population of coupled non-linear oscillators.In International Symposium on Mathematical Problems in Theoretical Physics: January 23–29, 1975, Kyoto University, Kyoto/Japan, pages 420–422. Springer, 1975.

[^78]: Daniel Lacker.Hierarchies, entropy, and quantitative propagation of chaos for mean field diffusions.Probability and Mathematical Physics, 4(2):377–432, 2023.

[^79]: Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.In International Conference on Learning Representations, 2020.

[^80]: Qianxiao Li, Long Chen, and Cheng Tai.Maximum principle based algorithms for deep learning.Journal of Machine Learning Research, 18:1–29, 2018.

[^81]: Wuchen Li.Hessian metric via transport information geometry.Journal of Mathematical Physics, 62(3), 03 2021.

[^82]: Hongzhou Lin and Stefanie Jegelka.Resnet with one-neuron hidden layers is a universal approximator.Advances in Neural Information Processing Systems, 31, 2018.

[^83]: Daniel Lacker and Luc Le Flem.Sharp uniform-in-time propagation of chaos.Probability Theory and Related Fields, pages 1–38, 2023.

[^84]: Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu.Understanding and improving transformer from a multi-particle dynamic system point of view.In International Conference on Learning Representations, 2020.

[^85]: Qianxiao Li, Ting Lin, and Zuowei Shen.Deep learning via dynamical systems: An approximation perspective.Journal of the European Mathematical Society, 25(5):1671–1709, 2022.

[^86]: Stanislaw Lojasiewicz.Une propriété topologique des sous-ensembles analytiques réels.Les équations aux dérivées partielles, 117:87–89, 1963.

[^87]: Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.A survey of transformers.AI Open, 2022.

[^88]: Shuyang Ling, Ruitu Xu, and Afonso S Bandeira.On the landscape of synchronization networks: A perspective from nonconvex optimization.SIAM Journal on Optimization, 29(3):1879–1907, 2019.

[^89]: MistralAI.[https://github.com/mistralai/mistral-finetune/blob/main/model/transformer.py](https://github.com/mistralai/mistral-finetune/blob/main/model/transformer.py), 2024.

[^90]: Javier Morales and David Poyato.On the trend to global equilibrium for Kuramoto oscillators.Annales de l’Institut Henri Poincaré C, 40(3):631–716, 2022.

[^91]: Sebastien Motsch and Eitan Tadmor.Heterophilious dynamics enhances consensus.SIAM Review, 56(4):577–621, 2014.

[^92]: Johan Markdahl, Johan Thunberg, and Jorge Gonçalves.Almost global consensus on the $n$ -sphere.IEEE Transactions on Automatic Control, 63(6):1664–1675, 2017.

[^93]: Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi.Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.Advances in Neural Information Processing Systems, 35:27198–27211, 2022.

[^94]: Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy.The shaped transformer: Attention models in the infinite depth-and-width limit.Advances in Neural Information Processing Systems, 36, 2024.

[^95]: OpenAI.[https://github.com/openai/gpt-2/blob/master/src/model.py](https://github.com/openai/gpt-2/blob/master/src/model.py), 2024.

[^96]: Felix Otto and Maria G Reznikoff.Slow motion of gradient flows.Journal of Differential Equations, 237(2):372–420, 2007.

[^97]: Felix Otto.The geometry of dissipative evolution equations: the porous medium equation.Communications in Partial Differential Equations, 26(1-2):101–174, 2001.

[^98]: Mary Phuong and Marcus Hutter.Formal algorithms for transformers.arXiv preprint arXiv:2207.09238, 2022.

[^99]: René Pinnau, Claudia Totzeck, Oliver Tse, and Stephan Martin.A consensus-based model for global optimization and its mean-field limit.Mathematical Models and Methods in Applied Sciences, 27(01):183–204, 2017.

[^100]: Domenec Ruiz-Balet and Enrique Zuazua.Neural ODE control for classification, approximation, and transport.SIAM Review, 65(3):735–773, 2023.

[^101]: Matthew Rosenzweig and Sylvia Serfaty.Global-in-time mean-field convergence for singular Riesz-type diffusive flows.The Annals of Applied Probability, 33(2):954–998, 2023.

[^102]: Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du.Token contrast for weakly-supervised semantic segmentation.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093–3102, 2023.

[^103]: Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré.Sinkformers: Transformers with doubly stochastic attention.In International Conference on Artificial Intelligence and Statistics, pages 3515–3530. PMLR, 2022.

[^104]: Sylvia Serfaty.Mean field limit for Coulomb-type flows.Duke Mathematical Journal, 169(15), 2020.

[^105]: Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet.On the empirical estimation of integral probability metrics.Electronic Journal of Statistics, 6:1550 – 1599, 2012.

[^106]: Michael Shub.Global stability of dynamical systems.Springer Science & Business Media, 2013.

[^107]: Steven H Strogatz.From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators.Physica D: Nonlinear Phenomena, 143(1-4):1–20, 2000.

[^108]: Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael Schaub.Residual connections and normalization can provably prevent oversmoothing in gnns.arXiv preprint arXiv:2406.02997, 2024.

[^109]: Gabor Szegö.Orthogonal polynomials, volume 23.American Mathematical Soc., 1939.

[^110]: Eitan Tadmor.Swarming: hydrodynamic alignment with pressure.Bulletin of the American Mathematical Society, 60(3):285–325, 2023.

[^111]: Yan Shuo Tan.Energy optimization for distributions on the sphere and improvement to the Welch bounds.Electronic Communications in Probability, 22:1 – 12, 2017.

[^112]: Richard Taylor.There is no non-zero stable fixed point for dense networks in the homogeneous Kuramoto model.Journal of Physics A: Mathematical and Theoretical, 45(5):055102, 2012.

[^113]: Paulo Tabuada and Bahman Gharesifard.Universal approximation power of deep residual neural networks through the lens of control.IEEE Transactions on Automatic Control, 2022.

[^114]: Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak.Transformers as support vector machines.Advances in Neural Information Processing Systems, 36, 2023.

[^115]: Alex Townsend, Michael Stillman, and Steven H Strogatz.Dense networks that do not synchronize and sparse ones that do.Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(8), 2020.

[^116]: James Vuckovic, Aristide Baratin, and Remi Tachet des Combes.A mathematical theory of attention.arXiv preprint arXiv:2007.02876, 2020.

[^117]: Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet.Novel type of phase transition in a system of self-driven particles.Physical Review Letters, 75(6):1226, 1995.

[^118]: Roman Vershynin.High-Dimensional Probability: An Introduction with Applications in Data Science.Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

[^119]: Cédric Villani.Limite de champ moyen.Cours de DEA, 2002:49, 2001.

[^120]: Cédric Villani.Optimal transport: old and new, volume 338.Springer, 2009.

[^121]: Tanya Veeravalli and Maxim Raginsky.Nonlinear controllability and function representation by neural stochastic differential equations.In Learning for Dynamics and Control Conference, pages 838–850. PMLR, 2023.

[^122]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017.

[^123]: Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie.On the role of attention masks and layernorm in transformers.arXiv preprint arXiv:2405.18781, 2024.

[^124]: Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie.Demystifying oversmoothing in attention-based graph neural networks.Advances in Neural Information Processing Systems, 36, 2024.

[^125]: James G Wendel.A problem in geometric probability.Mathematica Scandinavica, 11(1):109–111, 1962.

[^126]: Tong Xiao and Jingbo Zhu.Introduction to Transformers: an NLP Perspective.arXiv preprint arXiv:2311.17633, 2023.

[^127]: Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar.Are transformers universal approximators of sequence-to-sequence functions?In International Conference on Learning Representations, 2019.

[^128]: Aaron Zweig and Joan Bruna.A functional perspective on learning symmetric functions with neural networks.In International Conference on Machine Learning, pages 13023–13032. PMLR, 2021.

[^129]: Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz.Approximation capabilities of neural ODEs and invertible residual networks.In International Conference on Machine Learning, pages 11086–11095. PMLR, 2020.

[^130]: Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind.Stabilizing transformer training by preventing attention entropy collapse.In International Conference on Machine Learning, pages 40770–40803. PMLR, 2023.

[^131]: Haiteng Zhao, Shuming Ma, Dongdong Zhang, Zhi-Hong Deng, and Furu Wei.Are more layers beneficial to graph transformers?In The Eleventh International Conference on Learning Representations, 2023.

[^132]: Biao Zhang and Rico Sennrich.Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019.