{"notes":[{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"No Room Left","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"en","path":"/content/papers/en/graph-learning/over-squashing.146d06e9e263.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"problem-radius","text":"Problem Radius","depth":3},{"id":"the-rnn-bottleneck","text":"The RNN Bottleneck","depth":3},{"id":"looking-only-at-propagation-shares","text":"Looking Only at Propagation Shares","depth":3},{"id":"a-deliberately-constructed-task","text":"A Deliberately Constructed Task","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-fully-adjacent-final-layer","text":"A Fully Adjacent Final Layer","depth":3},{"id":"how-large-must-the-vector-be","text":"How Large Must the Vector Be?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"biological-datasets","text":"Biological Datasets","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"Smoothed Too Flat","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"en","path":"/content/papers/en/graph-learning/deeper-insights.ce733c5df590.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"compare-with-a-fully-connected-network-first","text":"Compare with a Fully Connected Network First","depth":3},{"id":"laplacian-smoothing","text":"Laplacian Smoothing","depth":3},{"id":"too-much-smoothing","text":"Too Much Smoothing","depth":3},{"id":"too-shallow-is-a-problem-too","text":"Too Shallow Is a Problem Too","depth":3},{"id":"a-remedy-give-gcn-more-labels-to-see","text":"A Remedy: Give GCN More Labels to See","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"GPS for Graphs","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"en","path":"/content/papers/en/graph-learning/graphgps.88c624c18f31.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"position-and-structure","text":"Position and Structure","depth":3},{"id":"why-message-passing-needs-them","text":"Why Message Passing Needs Them","depth":3},{"id":"message-passing-before-attention-can-lose-information-first","text":"Message Passing before Attention Can Lose Information First","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-ingredients","text":"Three Ingredients","depth":3},{"id":"one-gps-layer","text":"One GPS Layer","depth":3},{"id":"linear-complexity","text":"Linear Complexity","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"ablation-each-paths-contribution","text":"Ablation: Each Path's Contribution","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"reassessment-a-year-later","text":"Reassessment a Year Later","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"Since graph neural networks were introduced around 2005, distant nodes' information has been difficult to bring across.\n\nIn practice, GNNs stop improving beyond a few layers. The accepted explanation is over-smoothing: depth makes every node's representation alike. Uri Alon and Eran Yahav at Technion note that over-smoothing has mostly been observed on short-range tasks such as paper classification, whose answers mainly depend on nearby neighbors and need little depth. If a task truly requires distant information, something else may be going wrong.\n\n\n> [!TIP]\n> \n> For the origin and proof of over-smoothing, see our note:\n> \n> - **[[18.01] Deeper Insights: Smoothed Too Flat](/blog/papers/graph-learning/deeper-insights)**\n\n\nThey observe that nodes at distance $r$ need at least $r$ layers to interact. Yet each additional layer makes the neighborhood grow exponentially, squeezing all that information into one fixed-length vector. They name this **over-squashing**.\n\n## Defining the Problem\n\n### Problem Radius\n\nThe **problem radius** $r$ is the interaction distance a prediction task needs. Usually unknown beforehand, it is estimated in practice by tuning depth $K$.\n\nAfter $K$ layers, node $v$'s receptive field $\\mathcal N_v^K$ contains every node within $K$ hops. Long-range tasks require $K\\ge r$; otherwise distant nodes cannot reach the target. This is **under-reaching**.\n\nBut $|\\mathcal N_v^K|$ grows exponentially with $K$. Aggregation sums this growing information into a fixed-length vector, preventing important messages from reaching their destination. The model instead learns short-range training signals and generalizes poorly at test time.\n\n### The RNN Bottleneck\n\nThis recalls machine translation before attention. A seq2seq encoder compressed the whole input sentence into its final hidden state, losing information as sentences grew longer.\n\n[![Figure 1 of the paper: the left shows the RNN seq2seq bottleneck, passing input information along a chain to the final node. The right shows the GNN bottleneck: many tree-like nodes on the left funnel information layer by layer into the rightmost target, with red curves squeezed together just before it.](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)\n\n*Figure 1 from Alon and Yahav, taken from [**PDF page 2**](https://arxiv.org/pdf/2006.05205v4#page=2). The original caption is omitted.*\n\nThe authors argue that the GNN bottleneck is worse. RNN receptive fields grow **linearly** with steps; GNN fields grow **exponentially**. The same fixed-length vector must hold far more.\n\n### Looking Only at Propagation Shares\n\nTake a binary tree. Ignoring weights and nonlinearities, how much can each leaf contribute to the root after $r$ layers?\n\n```python\nimport numpy as np\n\ndef binary_tree(r):\n    n = 2 ** (r + 1) - 1                     # Root is 0; node i has children 2i+1 and 2i+2\n    A = np.zeros((n, n))\n    for i in range((n - 1) // 2):\n        for c in (2 * i + 1, 2 * i + 2):\n            A[i, c] = A[c, i] = 1\n    return A\n\nfor r in range(2, 8):\n    A = binary_tree(r) + np.eye(2 ** (r + 1) - 1)\n    d = A.sum(1)\n    A_hat = A / np.sqrt(np.outer(d, d))\n    share = np.linalg.matrix_power(A_hat, r)[0, -1]     # Final leaf's share at the root after r steps\n    print(f\"r={r}  Leaves {2 ** r:>3}, each leaf's share at the root {share:.5f}\")\n```\n\n```text\nr=2  Leaves   4, each leaf's share at the root 0.10206\nr=3  Leaves   8, each leaf's share at the root 0.02552\nr=4  Leaves  16, each leaf's share at the root 0.00638\nr=5  Leaves  32, each leaf's share at the root 0.00159\nr=6  Leaves  64, each leaf's share at the root 0.00040\nr=7  Leaves 128, each leaf's share at the root 0.00010\n```\n\nEach extra level doubles the leaves but shrinks each one's share at the root to roughly a quarter. At $r=7$, information from 128 leaves is mixed, each contributing around one ten-thousandth. This is only an intuitive demonstration; the paper's argument concerns how much information a vector can store.\n\n### A Deliberately Constructed Task\n\nTo isolate the bottleneck, the authors design **NeighborsMatch**.\n\nSeveral green nodes each have a different number of blue neighbors and a letter label. A target also has blue neighbors; its desired label is the letter of the green node with the same blue-neighbor count. The mapping differs in every graph, so all green-node information must reach the target for comparison each time. There is no shortcut.\n\nThe task looks contrived, but the authors compare it with a coding problem. A Python program declares several variables with types and values; choosing which variable belongs at a later position requires matching the needed type and value against visible variables. This is the VarMisuse task tested later.\n\n**Tree-NeighborsMatch** replaces the middle structure with a depth-$r$ binary tree. Green nodes are leaves, the target is the root, and all edges point toward the root. $r$ is the problem radius, directly controlling compression. Models use $r+1$ layers so information receives another nonlinearity after reaching the root.\n\n## Solving the Problem\n\nIdentifying and explaining over-squashing is the main contribution. The authors describe their repair as merely the simplest way to show that the problem is widespread and unaddressed.\n\n### A Fully Adjacent Final Layer\n\nReplace the final layer with a **fully adjacent (FA) layer**. Every pair of nodes is treated as adjacent, $\\mathcal N_v:=V$, keeping the layer type and adding no weights.\n\nThe first $K-1$ layers pass messages on the original graph and learn structurally informed node representations. The last layer lets all nodes interact directly, bypassing the bottleneck.\n\n### How Large Must the Vector Be?\n\nThe authors derive a combinatorial lower bound on hidden dimension $d$ for 100% training accuracy on Tree-NeighborsMatch. It grows factorially with $r$, faster than exponentially. With $d=32$, their experimental dimension, the radius can reach at most $r=7$. Practice is worse: gradient descent need not find an existing solution, and even $d=512$ fits only up to $r=7$ in their experiments.\n\n## Discussion\n\n### Tree-NeighborsMatch\n\n[![Figure 3 of the paper: problem radius r from 2 to 8 is on the horizontal axis, training accuracy on the vertical. GGNN, GAT, GIN and GCN all reach 1.0 at r=2,3. At r=4, GGNN and GAT remain at 1.0, GIN falls to 0.77 and GCN to 0.70; all then decline, falling below 0.2 at r=8.](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)\n\n*Figure 3 from Alon and Yahav, taken from [**PDF page 5**](https://arxiv.org/pdf/2006.05205v4#page=5). The original caption is omitted.*\n\nThis is **training** accuracy. GCN and GIN fit perfectly only through $r=3$; GCN falls to 70% at $r=4$. From $r=5$, none of the four GNNs fits the training set perfectly, despite seeing each example tens of thousands of times.\n\nGAT and GGNN last through $r=4$. The authors explain this through aggregation: GCN and GIN sum all neighboring messages into one vector before combining with the target's own representation. In their account, GAT's attention uses the target representation to select neighbors and pass relevant information upward. GGNN similarly filters through elementwise gates.\n\nIf none of these GNNs can fit training data here, how do they fit public long-range datasets? The authors hypothesize that models exploit short-range training signals and incidental features, failing to learn the long-range information lost in compression, and therefore generalizing poorly.\n\n### QM9\n\nQM9 has roughly 130,000 molecules, averaging around 18 atoms, with 13 quantum-chemistry regression targets. The authors use Brockschmidt's 2020 implementation, which searched 500 configurations for several GNNs. Most models' best depth is 8 layers, suggesting a need for long-range information.\n\nUsing the same code, splits and settings, they change only the last layer to FA and train each property 5 times. Error falls by 42% on average.\n\nThey then test alternative explanations:\n\n- **Under-reaching?** For at least 90% of examples, depth already equals or exceeds molecular diameter. Distant nodes can already reach one another. Ten-layer models do no better.\n- **Vectors too small?** Doubling dimensions improves only 5.5%; FA keeps the original dimension and adds no weights yet improves 42%.\n- **Must it be fully adjacent?** Adding only half the edges randomly in the last layer still reduces error by 31.5% on average. More added edges bring more improvement.\n- **Make every layer fully adjacent?** Error increases by 1500%. Graph structure remains necessary; information just needs a direct exchange at the end.\n\n### Biological Datasets\n\nNCI1 has 4,110 compounds labeled for anti-lung-cancer activity; ENZYMES has 600 enzymes in 6 classes. Following Errica et al.'s 2020 fair-comparison setup, the authors average 10 folds and 3 seeds, 30 runs in total.\n\nErrica et al. found that a graph-free baseline beat every GNN on ENZYMES. With FA, GIN improves by 8.1 percentage points there, beating that baseline by 2.5 points. On NCI1, GIN+FA beats the previous best GIN by 1.5 points. FA reduces average error by 12% on ENZYMES and 4.8% on NCI1.\n\nOn the VarMisuse code dataset, all five GNNs improve with FA on projects seen in training, with R-GCN reaching 88.4%. Most also improve on unseen projects: R-GCN rises from 82.9% to 83.8%, while GGNN and GNN-MLP decline slightly.\n\n## Conclusion\n\nThe paper identifies a previously unisolated cause of GNNs' long-range difficulty: exponentially growing receptive fields meet fixed-length vectors, squeezing information out at bottlenecks. Models such as GCN and GIN, which sum neighboring information indiscriminately, suffer most. A fully adjacent final layer with no additional weights substantially improves extensively tuned models.\n\nThe authors explicitly call FA a starting point. Later work widens bottlenecks by rewiring graphs or combines local message passing and global attention in every layer, as GraphGPS does.\n\n\n> [!TIP]\n> \n> For combining message passing and global attention in one layer, see our note:\n> \n> - **[[22.05] GraphGPS: GPS for Graphs](/blog/papers/graph-learning/graphgps)**\n\n\nIf your GNN stalls on a long-range task, try connecting all nodes in the last layer before increasing depth or width. No extra parameters are needed—just change that layer's adjacency matrix.\n","imageSizes":{"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png":{"width":1078,"height":356},"/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png":{"width":558,"height":316}}}