{"notes":[{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"GPS for Graphs","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"en","path":"/content/papers/en/graph-learning/graphgps.88c624c18f31.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"position-and-structure","text":"Position and Structure","depth":3},{"id":"why-message-passing-needs-them","text":"Why Message Passing Needs Them","depth":3},{"id":"message-passing-before-attention-can-lose-information-first","text":"Message Passing before Attention Can Lose Information First","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-ingredients","text":"Three Ingredients","depth":3},{"id":"one-gps-layer","text":"One GPS Layer","depth":3},{"id":"linear-complexity","text":"Linear Complexity","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"ablation-each-paths-contribution","text":"Ablation: Each Path's Contribution","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"reassessment-a-year-later","text":"Reassessment a Year Later","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]},{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"Fully Connected, Worse Results","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"en","path":"/content/papers/en/graph-learning/graph-transformer.c69ec318d0f9.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"attention-only-among-neighbors","text":"Attention Only among Neighbors","depth":3},{"id":"laplacian-positional-encoding","text":"Laplacian Positional Encoding","depth":3},{"id":"edge-features-and-normalization","text":"Edge Features and Normalization","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"what-happens-with-full-connectivity","text":"What Happens with Full Connectivity?","depth":3},{"id":"positional-encoding-and-batchnorm","text":"Positional Encoding and BatchNorm","depth":3},{"id":"comparison-with-gnns","text":"Comparison with GNNs","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"Hearing the Shape of a Graph","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"en","path":"/content/papers/en/graph-learning/san.2fbaa6b9ce67.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"learned-positional-encoding","text":"Learned Positional Encoding","depth":3},{"id":"full-connectivity-two-attention-mechanisms","text":"Full Connectivity, Two Attention Mechanisms","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-large-should-gamma-be","text":"How Large Should \\gamma Be?","depth":3},{"id":"contributions-of-connectivity-and-lpe","text":"Contributions of Connectivity and LPE","depth":3},{"id":"against-the-best-models-of-the-time","text":"Against the Best Models of the Time","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/graphormer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graphormer","subtitle":"Better Than You Might Think","author":"Z. Yuan","paperDate":"2021-06-09","publishedOn":"2024-03-28","world":"external/graphormer","paperTitle":"Do Transformers Really Perform Bad for Graph Representation?","paperUrl":"https://arxiv.org/abs/2106.05234","bibliography":{"title":"Do Transformers Really Perform Bad for Graph Representation?","authors":["Chengxuan Ying","Tianle Cai","Shengjie Luo","Shuxin Zheng","Guolin Ke","Di He","Yanming Shen","Tie-Yan Liu"],"url":"https://arxiv.org/abs/2106.05234","pdfUrl":"https://arxiv.org/pdf/2106.05234v5","codeUrl":"https://github.com/Microsoft/Graphormer","preprint":{"id":"2106.05234","submittedOn":"2021-06-09","url":"https://arxiv.org/abs/2106.05234"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html"},"citation":"@misc{graphlearninggraphormer2021,\n  title = {{Do Transformers Really Perform Bad for Graph Representation?}},\n  author = {Chengxuan Ying and Tianle Cai and Shengjie Luo and Shuxin Zheng and Guolin Ke and Di He and Yanming Shen and Tie-Yan Liu},\n  year = {2021},\n  eprint = {2106.05234},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.05234}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphormer.23b94dbe8ef3.bib"},"thumbnail":{"src":"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png","width":872,"height":620},"locale":"en","path":"/content/papers/en/graph-learning/graphormer.438265eee892.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"centrality-encoding","text":"Centrality Encoding","depth":3},{"id":"spatial-encoding","text":"Spatial Encoding","depth":3},{"id":"edge-encoding","text":"Edge Encoding","depth":3},{"id":"other-details","text":"Other Details","depth":3},{"id":"which-gnns-can-it-represent","text":"Which GNNs Can It Represent?","depth":3},{"id":"beyond-wl","text":"Beyond WL","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"pcqm4m-lsc","text":"PCQM4M-LSC","depth":3},{"id":"other-datasets","text":"Other Datasets","depth":3},{"id":"each-encodings-contribution","text":"Each Encoding's Contribution","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""}]}],"text":"In 2021, Graphormer wins the OGB-LSC quantum-chemistry challenge with a Transformer, bringing sudden attention to Graph Transformers.\n\nThe appeal is clear. Letting every node see every other seems to bypass message passing's old problems at once: over-smoothing with depth, over-squashing over distance and the WL ceiling on discrimination.\n\nBut Graph Transformers then each follow their own recipe. Some use Laplacian eigenvectors for positions, others shortest-path distances as attention biases, others message passing layers at the front. There is no common account of which choices help or why. Global attention's $O(N^2)$ cost also confines most models to graphs with a few hundred nodes.\n\n\n> [!TIP]\n> \n> For GT's neighbor-only attention with Laplacian encodings, SAN's fully connected dual attention, and Graphormer's shortest-path attention biases, see our notes:\n> \n> - **[[20.12] Graph Transformer: Fully Connected, Worse Results](/blog/papers/graph-learning/graph-transformer)**\n> - **[[21.06] SAN: Hearing the Shape of a Graph](/blog/papers/graph-learning/san)**\n> - **[[21.06] Graphormer: Better Than You Might Think](/blog/papers/graph-learning/graphormer)**\n\n\nLadislav Rampášek and Mikhail Galkin at Mila, Vijay Prakash Dwivedi at Nanyang Technological University and colleagues organize these choices into a recipe. GPS stands for General, Powerful, Scalable.\n\n## Defining the Problem\n\n### Position and Structure\n\nA Transformer cannot see the graph itself. Once all nodes are connected, it has no idea which were originally adjacent, so Graph Transformers must supply graph information separately.\n\nThe authors distinguish two kinds:\n\n- **Positional encoding (PE)** gives a node's position in the graph. Nearby nodes should have nearby encodings. Laplacian eigenvectors (LapPE) are common.\n- **Structural encoding (SE)** describes node or graph structure. Nodes with similar surrounding subgraphs should have similar encodings. A common choice is the diagonal of the random-walk matrix's $m$th power (RWSE): the probability of returning to self after $m$ steps.\n\nEach splits further into local, global and relative forms. Global SE could be Laplacian eigenvalues; relative SE could indicate whether two nodes share a substructure. The authors tabulate existing approaches and argue that PE and SE should preferably be node features or real edges, allowing linear-complexity attention.\n\n### Why Message Passing Needs Them\n\nStandard message passing is as discriminative as the 1-WL test. The authors illustrate with Circular Skip Link graphs: two non-isomorphic graphs get identical WL colors at every node, so no message passing network distinguishes them. Global PE gives unique starting colors and separates them; local RWSE can also detect their different skip connections.\n\nThese signals cannot be learned from structure through message passing alone; they must be provided separately.\n\nTake a smaller example: a six-node hexagon and two disjoint triangles. Every node has two neighbors, so WL fails. Compute RWSE:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=3, suppress=True)\n\ndef graph(edges, n=6):\n    A = np.zeros((n, n))\n    for i, j in edges:\n        A[i, j] = A[j, i] = 1\n    return A\n\nhexagon = graph([(0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 0)])\ntwo_triangles = graph([(0, 1), (1, 2), (2, 0), (3, 4), (4, 5), (5, 3)])\n\ndef rwse(A, steps=4):\n    P = A / A.sum(1, keepdims=True)                  # Random-walk transition matrix D^-1 A\n    return np.stack([np.diag(np.linalg.matrix_power(P, m)) for m in range(1, steps + 1)], axis=1)\n\nprint(\"Hexagon      node 0:\", rwse(hexagon)[0])\nprint(\"Two triangles  node 0:\", rwse(two_triangles)[0])\n```\n\n```text\nHexagon      node 0: [0.    0.5   0.    0.375]\nTwo triangles  node 0: [0.    0.5   0.25  0.375]\n```\n\nThe difference is the third value, the three-step return probability: 0.25 on triangles, 0 on the hexagon, which has no length-3 cycle. The authors also note that for odd $m$, RWSE can detect whether a node lies on a cycle of length $m$.\n\n### Message Passing before Attention Can Lose Information First\n\nA common architecture, such as GraphTrans, stacks message passing before a Transformer. The authors see a risk: the initial layers' over-smoothing, over-squashing and expressiveness limits may discard information that no later Transformer can recover.\n\nConvolution before a Transformer works in vision because grid convolutions are highly expressive; the authors question whether the same reasoning holds on graphs.\n\n## Solving the Problem\n\n### Three Ingredients\n\nGPS has three parts:\n\n1. **Positional/structural encodings**, added to node or edge features at input.\n2. **Local message passing**, following real edges and handling edge features.\n3. **Global attention**, letting every node see all nodes.\n\nEach is replaceable. Message passing can use GINE, GatedGCN or PNA; global attention can use standard Transformer, Performer or BigBird.\n\n[![Figure 1 of the paper: the modular GPS architecture. Three columns on the left process positional encoding, structural encoding and graph features through MLP, DeepSet, SignNet or other components and add them to node and edge features. On the right, each of L GPS layers runs any message passing layer (GINE, GatedGCN, PNA) and any global attention (Transformer, Performer, BigBird) in parallel, sums them and applies a two-layer MLP.](/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png)](/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png)\n\n*Figure 1 from Rampášek et al., taken from [**PDF page 3**](https://arxiv.org/pdf/2205.12454v4#page=3). The original caption is omitted.*\n\n### One GPS Layer\n\nEach layer runs two parallel paths, sums their outputs, then applies an MLP:\n\n$$\nX_M^{\\ell+1},\\,E^{\\ell+1}=\\mathrm{MPNN}_e^{\\ell}\\left(X^\\ell,E^\\ell,A\\right)\n$$\n\n$$\nX_T^{\\ell+1}=\\mathrm{GlobalAttn}^{\\ell}\\left(X^\\ell\\right)\n$$\n\n$$\nX^{\\ell+1}=\\mathrm{MLP}^{\\ell}\\left(X_M^{\\ell+1}+X_T^{\\ell+1}\\right)\n$$\n\nEdge features enter only the message passing path. Residuals and BatchNorm are omitted from the formulas but appear on both paths in the implementation.\n\nUsing the two graphs above, write one randomly weighted GPS layer. Message passing simply sums neighbors, global attention is single-head Transformer attention, and the final sum over all nodes gives the graph representation:\n\n```python\nrng = np.random.default_rng(0)\nd = 4\nW_in = rng.normal(size=(4, d))                       # Project input features to d dimensions\nWm, Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(4))\nW1, W2 = rng.normal(size=(d, 2 * d)), rng.normal(size=(2 * d, d))\n\ndef gps_layer(X, A):\n    X_M = X + np.maximum((A @ X) @ Wm, 0)            # Local: neighbor-sum message passing with a residual\n    Q, K, V = X @ Wq, X @ Wk, X @ Wv\n    S = Q @ K.T / np.sqrt(d)\n    att = np.exp(S - S.max(1, keepdims=True))\n    att /= att.sum(1, keepdims=True)                 # Global: every node sees every other node\n    X_T = X + att @ V\n    H = X_M + X_T                                    # Add both paths\n    return H + np.maximum(H @ W1, 0) @ W2            # Then apply an MLP\n\ndef embed(A, use_rwse):\n    feats = rwse(A) if use_rwse else np.ones((len(A), 4))\n    return gps_layer(feats @ W_in, A).sum(0)        # sum readout\n\nfor use in (False, True):\n    same = np.allclose(embed(hexagon, use), embed(two_triangles, use))\n    print(\"With RWSE\" if use else \"Without encoding\", \"→ Same graph representations:\", same)\n```\n\n```text\nWithout encoding → Same graph representations: True\nWith RWSE → Same graph representations: False\n```\n\nPause at this result. Without encodings, even global attention cannot separate the graphs. Every input is identical, so attention distributes uniformly; averaging over all nodes and averaging over neighbors give the same result. With RWSE, inputs already reveal whether nodes lie on triangles, and the same GPS layer separates them.\n\n### Linear Complexity\n\nMessage passing costs $O(E)$. Standard global attention costs $O(N^2)$; replacing it with linear attention such as Performer gives $O(N)$. Because PE and SE reside on nodes and real edges, and edge features never enter global attention, no full attention matrix is required. On sparse molecular or knowledge graphs, edges grow proportionally with nodes, making the model $O(N)$ overall. Laplacian eigendecomposition preprocessing is excluded.\n\nIn practice, a roughly 6-million-parameter model takes 196 seconds per epoch on ogbg-molpcba, versus SAN's 883 seconds on the same GPU type.\n\nThe authors also prove that, with suitable components, the architecture preserves edge information and is a universal function approximator on graphs.\n\n## Discussion\n\n### Ablation: Each Path's Contribution\n\nThe authors dissect components on four datasets:\n\n| Setting | ZINC MAE ↓ | PCQM4Mv2 subset MAE ↓ | CIFAR10 accuracy ↑ | MalNet-Tiny accuracy ↑ |\n| --- | ---: | ---: | ---: | ---: |\n| Full GPS (Transformer) | 0.070 | 0.1159 | 72.31% | 93.50% |\n| Performer global attention | 0.071 | 0.1142 | 70.67% | 92.64% |\n| BigBird global attention | 0.071 | 0.1237 | 70.48% | 92.34% |\n| Remove global attention | 0.070 | 0.1213 | 69.95% | 92.23% |\n| Remove message passing | 0.217 | 0.3294 | 68.86% | 73.90% |\n\nRemoving message passing badly hurts all four, dropping MalNet-Tiny from 93.50% to 73.90%. The authors regard it as indispensable because without it edge features have no entry point.\n\nGlobal attention's contribution depends on the dataset. ZINC stays at 0.070 with or without it: the target molecular property depends only on counts of local substructures, so global links are unnecessary, but structural encoding matters greatly. Transformer attention helps the other three. Performer trails the standard Transformer but scales to large graphs; BigBird brings no significant gain here and is slower than Performer.\n\nFor encodings, ZINC MAE is 0.113 without any, 0.070 with RWSE, and 0.116 with LapPE. RWSE gives consistent gains at relatively low cost; SignNet is the best single encoding but costs more.\n\n### Comparison with Other Models\n\nOn Benchmarking GNNs' five datasets, GPS reaches ZINC MAE 0.070, below GCN's 0.367, GIN's 0.526 and Graphormer's 0.122; CIFAR10 accuracy is 72.30%. It places second on MNIST and CLUSTER, fourth on PATTERN, only a few tenths of a percentage point behind the leaders.\n\nOn OGB graph-level tasks, GPS is top three on ogbg-molpcba, ogbg-ppa and ogbg-code2, beating all other Graph Transformers according to the authors. On PCQM4Mv2, GPS-medium with 19.4 million parameters has validation MAE 0.0858, slightly better than Graphormer's 0.0864 with 48.3 million.\n\nOn the Long Range Graph Benchmark (LRGB), GPS combines GatedGCN, Transformer and LapPE and beats every baseline on four of five datasets. Peptides-func average precision is 0.6535 versus GCN's 0.5930; PascalVOC-SP F1 is 0.3748 versus 0.1268.\n\n### Reassessment a Year Later\n\nThe LRGB gap is later reexamined. In September 2023, Tönshoff et al. at RWTH Aachen perform basic hyperparameter searches for baselines including GCN. Peptides-func GCN rises to 0.6860, above the GPS rerun's 0.6534. The most important hyperparameter is the prediction head: replacing LRGB's single linear layer with a two-layer MLP closes most of the gap, while GPS barely changes. [Tönshoff et al.](https://arxiv.org/abs/2309.00367v2)\n\nIn 2025, Luo et al. add six techniques, including residuals, normalization, FFNs and positional encoding, to GCN, GIN and GatedGCN. Across 14 graph-level datasets, they match or beat Graph Transformers such as GPS.\n\nLooking back, several GPS ingredients—positional and structural encoding, residuals, feedforward layers—later also appear among the techniques that let classic GNNs catch up.\n\n\n> [!TIP]\n> \n> For over-squashing, the five LRGB datasets, and both reassessments, see our notes:\n> \n> - **[[20.06] Over-squashing: No Room Left](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.06] LRGB: Tests from Afar](/blog/papers/graph-learning/lrgb)**\n> - **[[23.09] Where Did the Gap Go?: Where Did the Gap Go?](/blog/papers/graph-learning/where-did-the-gap-go)**\n> - **[[25.02] GNN+: Old Steeds, New Ambitions](/blog/papers/graph-learning/gnn-plus)**\n\n\n## Conclusion\n\nGraphGPS separates Graph Transformers into three replaceable ingredients: positional/structural encoding, local message passing and global attention. Removing message passing hurts across the board; removing global attention hurts according to the task's need for distant information. Without encodings, attention still cannot see structures invisible to message passing.\n\nIf there is one thing to take away, it may be RWSE. A few sparse matrix multiplications produce an inexpensive addition to any GNN's input.\n\nAt least on ZINC, removing global attention leaves MAE unchanged; removing RWSE worsens it from 0.070 to 0.113. A few random-walk return probabilities do more than a whole global-attention layer.\n","imageSizes":{"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png":{"width":1116,"height":798}}}