{"notes":[{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"Hearing the Shape of a Graph","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"en","path":"/content/papers/en/graph-learning/san.2fbaa6b9ce67.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"learned-positional-encoding","text":"Learned Positional Encoding","depth":3},{"id":"full-connectivity-two-attention-mechanisms","text":"Full Connectivity, Two Attention Mechanisms","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-large-should-gamma-be","text":"How Large Should \\gamma Be?","depth":3},{"id":"contributions-of-connectivity-and-lpe","text":"Contributions of Connectivity and LPE","depth":3},{"id":"against-the-best-models-of-the-time","text":"Against the Best Models of the Time","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"Fully Connected, Worse Results","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"en","path":"/content/papers/en/graph-learning/graph-transformer.c69ec318d0f9.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"attention-only-among-neighbors","text":"Attention Only among Neighbors","depth":3},{"id":"laplacian-positional-encoding","text":"Laplacian Positional Encoding","depth":3},{"id":"edge-features-and-normalization","text":"Edge Features and Normalization","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"what-happens-with-full-connectivity","text":"What Happens with Full Connectivity?","depth":3},{"id":"positional-encoding-and-batchnorm","text":"Positional Encoding and BatchNorm","depth":3},{"id":"comparison-with-gnns","text":"Comparison with GNNs","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]}],"text":"In 1966, mathematician Mark Kac writes “Can One Hear the Shape of a Drum?” A drum's shape determines its vibration frequencies; conversely, can those frequencies alone reveal its shape?\n\nGraphs have frequencies too: their Laplacian eigenvalues and eigenvectors. Devin Kreuzer, Dominique Beaini, William L. Hamilton and colleagues at McGill, Mila and Valence Discovery want a Transformer to “hear” these frequencies and learn each node's position.\n\nThey also want to overturn a verdict. Six months earlier, Graph Transformer (GT) tried letting every node see every other, badly hurting all three datasets and leading its authors to favor neighbor-only attention. SAN, Spectral Attention Network, aims to show that full connectivity can work.\n\n\n> [!TIP]\n> \n> For GT's sparse attention, Laplacian positional encodings and full-connectivity experiments, see our note:\n> \n> - **[[20.12] Graph Transformer: Fully Connected, Worse Results](/blog/papers/graph-learning/graph-transformer)**\n\n\n## Defining the Problem\n\nThe authors list message passing's known limitations: WL-bounded graph discrimination, over-smoothing with depth, and over-squashing over distance. All are tied to graph structure directly determining computational routes.\n\nA Transformer lets all positions interact and supplies structure softly, through positional encoding. For graphs, that encoding is the key.\n\nGT already uses Laplacian eigenvectors, but the authors identify unresolved issues:\n\n- **Eigenvalues carry information too.** GT supplies eigenvectors without eigenvalues. Sequence frequencies are fixed, so order identifies them; graph frequencies vary, and order alone loses the frequency corresponding to each vector.\n- **Repeated eigenvalues.** With multiplicity, any combination within the corresponding eigenspace is a valid eigenvector.\n- **Variable counts.** A graph has at most as many eigenvectors as nodes. Taking a fixed $k$ means using a smaller fraction on larger graphs.\n- **Signs.** $k$ vectors have $2^k$ sign combinations. Random sign flips are manageable for small $k$, less so as it grows.\n\nRepeated eigenvalues appear even in two triangles joined by a bridge. A, B, C are on the left, D, E, F on the right, with bridge C–D. Its normalized Laplacian has eigenvalue 1.5 twice:\n\n```python\nimport numpy as np\n\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)]\nA = np.zeros((6, 6))\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\nd = A.sum(1)\nL = np.eye(6) - A / np.sqrt(np.outer(d, d))\nlam = np.clip(np.linalg.eigvalsh(L), 0, None)     # Eigenvalues are nonnegative; clip floating-point error\nprint(\"Eigenvalues:\", lam.round(2))\n\nu1 = np.array([1, -1, 0, 0, 0, 0]) / np.sqrt(2)   # Nonzero only at A and B in the left triangle\nu2 = np.array([0, 0, 0, 0, 1, -1]) / np.sqrt(2)   # Nonzero only at E and F in the right triangle\nmix = (u1 + u2) / np.sqrt(2)                      # A combination of both\nfor name, v in [(\"u1  \", u1), (\"u2  \", u2), (\"Mix\", mix)]:\n    print(name, \"is an eigenvector for eigenvalue 1.5:\", np.allclose(L @ v, 1.5 * v), \"  Entry at A:\", round(v[0], 2))\n```\n\n```text\nEigenvalues: [0.   0.2  1.17 1.5  1.5  1.63]\nu1   is an eigenvector for eigenvalue 1.5: True   Entry at A: 0.71\nu2   is an eigenvector for eigenvalue 1.5: True   Entry at A: 0.0\nMix is an eigenvector for eigenvalue 1.5: True   Entry at A: 0.5\n```\n\nAll three vectors are valid, yet A gets 0.71, 0 or 0.5 depending on the library's choice. Using that entry directly as A's position could make learning depend on the library.\n\n## Solving the Problem\n\nSAN has two stages: learn a positional encoding for each node, concatenate it with node features, and feed the result into a fully connected Transformer.\n\n[![Original teaching diagram: on the left, two triangles joined at C–D have 7 original solid edges and 8 added dashed blue connections. Original-edge weights multiply by 1/(1+γ), added connections by γ/(1+γ); γ=0 uses only original edges, γ=1 treats both equally. On the right, node j's m eigenvalue/eigenvector-entry pairs pass through a linear layer, Transformer and sum to form LPE, concatenated to node features.](/content/papers/assets/5e199d50e58434e7be5a6ef7ef8fb95442259ce60cab7830935787e402e13ca3.png)](/content/papers/assets/5e199d50e58434e7be5a6ef7ef8fb95442259ce60cab7830935787e402e13ca3.png)\n\n*Original diagram based on the formulas in §3.1 and §3.3 of the paper.*\n\n### Learned Positional Encoding\n\nThe right half is SAN's learned positional encoding (LPE).\n\nFor node $j$, take the $m$ smallest eigenvalues and pair each $\\lambda_i$ with the corresponding eigenvector entry $\\phi_{i,j}$, producing a length-$m$ sequence. The authors view eigenvectors as sitting on a frequency axis, with eigenvalues specifying positions along it.\n\nA linear layer maps each pair to $k$ dimensions. A Transformer encoder attends across the $m$ elements, then sums them into node $j$'s $k$-dimensional LPE. If the graph has fewer than $m$ nodes, masked padding fills the remainder.\n\nThis addresses three of the four issues. Eigenvalues accompany eigenvectors; sequence lengths can vary, with $m$ even reaching the largest graph size in the dataset to use the full spectrum; and the Transformer can potentially combine or ignore vectors from repeated eigenspaces. Signs remain unresolved: SAN randomly flips them during training, like GT.\n\nThe appendix also describes an edge-wise version, using sign-invariant quantities such as $|\\phi_{i,j_1}-\\phi_{i,j_2}|$ and $\\phi_{i,j_1}\\phi_{i,j_2}$ as pairwise features. It is theoretically stronger but costs up to $O(N^4)$ with full connectivity, so the main text uses node-wise LPE.\n\n### Full Connectivity, Two Attention Mechanisms\n\nThe left half is the main Transformer. Every nonadjacent pair gets an added connection with its own edge embedding. Original edges use $Q^1$, $K^1$, $E^1$; added connections use $Q^2$, $K^2$, $E^2$. Hyperparameter $\\gamma$ balances their weights:\n\n$$\nw_{ij}=\n\\begin{cases}\n\\dfrac{1}{1+\\gamma}\\cdot\\mathrm{softmax}\\big(\\hat w_{ij}\\big) & i,j\\ \\text{originally adjacent}\\\\[2mm]\n\\dfrac{\\gamma}{1+\\gamma}\\cdot\\mathrm{softmax}\\big(\\hat w_{ij}\\big) & \\text{other node pairs}\n\\end{cases}\n$$\n\nAt $\\gamma=0$, added connections have no weight and attention is neighbor-only; at $\\gamma=1$, neither set is favored. This knob trades local structure against distant interaction. Residuals, normalization and feedforward layers follow as in GT. Edge representations are not updated: the authors say doing so adds complexity with almost no benefit.\n\n## Discussion\n\n### How Large Should $\\gamma$ Be?\n\nThe authors sweep $\\gamma$ on ZINC, PATTERN, CLUSTER and MolHIV. On molecular datasets ZINC and MolHIV, the best $\\log\\gamma$ lies between −7 and −5, giving added connections only a tiny share. PATTERN and CLUSTER prefer larger $\\gamma$.\n\nThe explanation is that molecular graphs are small and depend on local structures such as rings and particular bonds; a few sparse-attention layers already span the graph. CLUSTER is semi-supervised, with some true node labels provided in each graph; full connectivity exposes those labeled nodes to everyone at every layer.\n\n### Contributions of Connectivity and LPE\n\nKeeping $\\gamma$ fixed and parameters around 500,000, the authors ablate both designs:\n\n| Attention | LPE | ZINC (MAE ↓) | PATTERN | CLUSTER | MolHIV (ROC-AUC) |\n| --- | --- | ---: | ---: | ---: | ---: |\n| Sparse | No | 0.267 | 83.61% | 75.68% | 73.46% |\n| Sparse | Yes | 0.198 | 81.33% | 75.74% | 76.61% |\n| Fully connected | No | 0.392 | 86.32% | 76.45% | 73.84% |\n| Fully connected | Yes | **0.157** | **86.44%** | **76.69%** | **77.57%** |\n\nBoth factors matter. On ZINC, full connectivity without LPE worsens MAE from 0.267 to 0.392, matching GT's observation; with LPE, it becomes best at 0.157. LPE helps molecular datasets most, which the authors connect to substructure detection; PATTERN and CLUSTER are simpler clustering tasks, so its effect is smaller.\n\n### Against the Best Models of the Time\n\nTuned SAN reaches ZINC MAE 0.139, ahead of PNA's 0.142. PATTERN at 86.58% and CLUSTER at 76.69% also lead the table. Fully connected GT has ZINC MAE 0.598 in the same table. To the authors' knowledge, SAN is the first fully connected model to perform well on graph benchmarks.\n\nOGB molecular results are less striking. MolHIV test ROC-AUC is 77.85%, below DGN's 79.70%. The authors note SAN's validation score of 85.30% exceeds PNA's 84.25% and DGN's 84.70%, attributing the outcome to low validation/test correlation on this dataset. MolPCBA average precision is 27.65%, versus DGN's 28.85%.\n\n### The Cost\n\nFull attention costs $O(N^2)$, LPE $O(m^2N)$, far more than message passing's $O(|E|)$, as the authors acknowledge. Later work makes the cost clearer: GraphGPS reports 196 seconds per epoch for roughly 6 million-parameter GPS on ogbg-molpcba, versus 883 seconds for SAN on the same GPU type. On LRGB's COCO-SP, with over 120,000 graphs averaging nearly 480 nodes, SAN fails to converge after 60 hours on an A100.\n\n## Conclusion\n\nSAN demonstrates that fully connected Transformers can perform well on graphs, provided positional encodings carefully handle the spectrum and original versus added edges score separately, balanced by $\\gamma$. Its ablation also explains GT's observation: full connectivity without good positional encoding makes things worse.\n\nThe following year, GraphGPS, also involving Beaini, treats this attention as a replaceable module alongside message passing in each layer. It also offers encodings such as SignNet specifically for sign ambiguity.\n\nFor graphs with only tens of nodes, SAN is worth trying. As graphs grow, quadratic computation becomes the first obstacle.\n","imageSizes":{"/content/papers/assets/5e199d50e58434e7be5a6ef7ef8fb95442259ce60cab7830935787e402e13ca3.png":{"width":1280,"height":840}}}