{"notes":[{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"Fully Connected, Worse Results","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"en","path":"/content/papers/en/graph-learning/graph-transformer.c69ec318d0f9.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"attention-only-among-neighbors","text":"Attention Only among Neighbors","depth":3},{"id":"laplacian-positional-encoding","text":"Laplacian Positional Encoding","depth":3},{"id":"edge-features-and-normalization","text":"Edge Features and Normalization","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"what-happens-with-full-connectivity","text":"What Happens with Full Connectivity?","depth":3},{"id":"positional-encoding-and-batchnorm","text":"Positional Encoding and BatchNorm","depth":3},{"id":"comparison-with-gnns","text":"Comparison with GNNs","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"Hearing the Shape of a Graph","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"en","path":"/content/papers/en/graph-learning/san.2fbaa6b9ce67.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"learned-positional-encoding","text":"Learned Positional Encoding","depth":3},{"id":"full-connectivity-two-attention-mechanisms","text":"Full Connectivity, Two Attention Mechanisms","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-large-should-gamma-be","text":"How Large Should \\gamma Be?","depth":3},{"id":"contributions-of-connectivity-and-lpe","text":"Contributions of Connectivity and LPE","depth":3},{"id":"against-the-best-models-of-the-time","text":"Against the Best Models of the Time","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"GPS for Graphs","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"en","path":"/content/papers/en/graph-learning/graphgps.88c624c18f31.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"position-and-structure","text":"Position and Structure","depth":3},{"id":"why-message-passing-needs-them","text":"Why Message Passing Needs Them","depth":3},{"id":"message-passing-before-attention-can-lose-information-first","text":"Message Passing before Attention Can Lose Information First","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-ingredients","text":"Three Ingredients","depth":3},{"id":"one-gps-layer","text":"One GPS Layer","depth":3},{"id":"linear-complexity","text":"Linear Complexity","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"ablation-each-paths-contribution","text":"Ablation: Each Path's Contribution","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"reassessment-a-year-later","text":"Reassessment a Year Later","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"When a Transformer processes a sentence, every word looks at every other word. Viewed another way, the sentence is a fully connected graph: words are nodes, with an edge between every pair.\n\nIf sentences are already graphs, can real graphs be fed straight into a Transformer?\n\nVijay Prakash Dwivedi and Xavier Bresson at Nanyang Technological University argue against a direct transplant. A sentence has no given connectivity and usually only tens or hundreds of words, making full connectivity reasonable and affordable. A graph brings its own structure and may have millions of nodes; connecting everything wastes that structure and costs too much.\n\nTheir late-2020 Graph Transformer (GT below) is an early attempt to generalize Transformers to arbitrary graphs. SAN six months later, and GraphGPS another year later, both use it as a starting comparison.\n\n\n> [!TIP]\n> \n> For SAN's later revival of full connectivity and GraphGPS's combination of both approaches in one layer, see our notes:\n> \n> - **[[21.06] SAN: Hearing the Shape of a Graph](/blog/papers/graph-learning/san)**\n> - **[[22.05] GraphGPS: GPS for Graphs](/blog/papers/graph-learning/graphgps)**\n\n\n## Defining the Problem\n\nThe authors split the problem into two parts.\n\n**Sparsity.** Earlier graph Transformers, such as Li et al. (2019), let every node see all nodes for global information. The authors argue that this discards sparse graph connectivity, a valuable inductive bias. Positional encodings or virtual nodes could supply global information instead.\n\n**Positional encoding.** Sentence words have an order, encoded by sinusoids in a Transformer. Graphs have no canonical node order and can have symmetries, making unique positions hard to define. The authors see missing positional information as one reason for GAT's limited performance despite neighbor-only attention.\n\nThe paper tests whether neighbor-only attention with suitable positional encoding can produce a general, effective graph Transformer.\n\n## Solving the Problem\n\n[![Figure 1 of the paper: the left diagram is a Graph Transformer layer. Node features with Laplacian positional encoding λ undergo Q, K, V projections, multiplication, scaling, softmax and summation, then Add&Norm, FFN and Add&Norm, repeated L times. The edge-feature version on the right adds e_ij: attention scores multiply E-projected edge features, and edge representations have their own FFN and normalization.](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)\n\n*Figure 1 from Dwivedi and Bresson, taken from [**PDF page 2**](https://arxiv.org/pdf/2012.09699v2#page=2). The original caption is omitted.*\n\nThe left is the basic version, the right adds edge features. There are three changes from the original Transformer.\n\n### Attention Only among Neighbors\n\nOne GT layer's attention is almost identical to Transformer attention, differing only in the summation range:\n\n$$\n\\hat h_i^{\\ell+1}=O_h^\\ell\\,\\Big\\Vert_{k=1}^{H}\\Big(\\sum_{j\\in\\mathcal N_i}w_{ij}^{k,\\ell}V^{k,\\ell}h_j^\\ell\\Big),\\qquad\nw_{ij}^{k,\\ell}=\\mathrm{softmax}_j\\Big(\\frac{Q^{k,\\ell}h_i^\\ell\\cdot K^{k,\\ell}h_j^\\ell}{\\sqrt{d_k}}\\Big)\n$$\n\n$\\mathcal N_i$ is node $i$'s neighborhood, and $\\Vert$ concatenates $H$ heads. The Transformer sums over all positions, GT only over neighbors. Residuals, normalization and a feedforward network follow as usual.\n\n### Laplacian Positional Encoding\n\nPositional encoding uses Laplacian eigenvectors:\n\n$$\n\\Delta=I-D^{-1/2}AD^{-1/2}=U^\\top\\Lambda U\n$$\n\nThe authors take the $k$ smallest nontrivial eigenvectors, skipping the one with eigenvalue 0. Each node's entries in these vectors form its positional encoding $\\lambda_i$. After a linear projection, it is added to node features only at the input, not at intermediate layers.\n\nWhy eigenvectors? They generalize NLP's sinusoidal encodings to graphs, the authors argue, bringing distance information so nearby nodes get nearby encodings.\n\nTry two triangles joined by a bridge: A, B, C on the left, D, E, F on the right, C–D as the bridge. Take $k=2$:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\nnames = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)]\nA = np.zeros((6, 6))\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\n\nd = A.sum(1)\nL = np.eye(6) - A / np.sqrt(np.outer(d, d))      # Δ = I - D^-1/2 A D^-1/2\nlam, U = np.linalg.eigh(L)                      # Eigenvalues in ascending order\nU = U * np.sign(U[0])                           # Our convention: make A's entry positive\n\npe = U[:, 1:3]                                  # Skip eigenvalue 0 and take the next 2 vectors\nprint(\"Eigenvalues:\", lam[1:3])\nfor name, row in zip(names, pe):\n    print(name, row)\n```\n\n```text\nEigenvalues: [0.2  1.17]\nA [0.45 0.33]\nB [0.45 0.33]\nC [ 0.32 -0.53]\nD [-0.32 -0.53]\nE [-0.45  0.33]\nF [-0.45  0.33]\n```\n\nThe first dimension separates left and right, positive on one triangle and negative on the other. The second separates bridge nodes C and D from the four outer nodes. A and B receive identical encodings because they are structurally symmetric; structure alone cannot distinguish them.\n\nEigenvectors have another complication: arbitrary signs. The `np.sign` line is our own convention; multiplying the whole vector by −1 leaves a valid eigenvector:\n\n```python\nv = pe[:, 0]\nprint(\"Original vector:\", np.allclose(L @ v, lam[1] * v))\nprint(\"After sign flip  :\", np.allclose(L @ -v, lam[1] * -v))\n```\n\n```text\nOriginal vector: True\nAfter sign flip  : True\n```\n\nThe same graph can yield opposite encodings, with $2^k$ sign combinations for $k$ vectors. On another graph, a convention such as making the first node positive may lack justification. GT randomly flips signs during training so the model learns to ignore them.\n\n### Edge Features and Normalization\n\nMolecular bond types and knowledge-graph relations live on edges. GT's edge-feature version first computes neighboring attention scores, multiplies them by projected edge features $E^{k,\\ell}e_{ij}^\\ell$, then applies softmax. Edge representations also have their own feedforward networks and update layer by layer.\n\nThe final change replaces Transformer LayerNorm with BatchNorm, giving faster training and better generalization.\n\n## Discussion\n\n### Experimental Setup\n\nThe authors use three Benchmarking GNNs datasets:\n\n- **ZINC**: molecular graph regression on a 12,000-molecule subset, predicting a solubility-related property with MAE, lower being better. Bond types are edge features, so the edge-feature GT is used.\n- **PATTERN**: 14,000 stochastic-block-model graphs, with binary node classification.\n- **CLUSTER**: 12,000 generated graphs, assigning each node to one of 6 clusters.\n\nModels have 10 layers, 8 attention heads and around 500,000 parameters. Each experiment uses 4 random seeds.\n\n### What Happens with Full Connectivity?\n\nThe authors run the same model on the original graph and a fully connected graph. With BatchNorm and Laplacian encodings:\n\n| Dataset | Neighbors only | All nodes |\n| --- | ---: | ---: |\n| ZINC (MAE ↓) | 0.226 | 0.598 |\n| CLUSTER (accuracy ↑) | 73.17% | 27.12% |\n| PATTERN (accuracy ↑) | 84.81% | 54.94% |\n\nAll three deteriorate dramatically. PATTERN has two classes, so 54.94% is near chance; CLUSTER has 6, so 27.12% is only modestly above chance at 16.7%. The fully connected ZINC version omits edge features, deliberately testing a setting with no sparse structure.\n\nThe authors conclude that sparse connectivity is a crucial inductive bias on arbitrary graphs.\n\n### Positional Encoding and BatchNorm\n\nAcross four combinations of LayerNorm/BatchNorm and with/without positional encoding, BatchNorm plus Laplacian encoding wins on all three datasets. Some settings vary enormously: on PATTERN, LayerNorm without encodings gives 73.14% ± 13.63% across four seeds.\n\nThe authors also compare Graph-BERT's WL positional encodings:\n\n| Positional encoding | ZINC (MAE ↓) | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| None | 0.264 | 72.14% | 83.95% |\n| Laplacian | **0.226** | **73.17%** | **84.81%** |\n| WL (Graph-BERT) | 0.267 | 70.79% | 75.49% |\n\nWL encoding reaches 97.03% training accuracy on PATTERN but only 75.49% test accuracy; the authors suspect overfitting on generated graphs.\n\n### Comparison with GNNs\n\nFinally, GNNs under the same 500,000-parameter budget:\n\n| Model | ZINC (MAE ↓) | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| GCN | 0.367 | 68.50% | 71.89% |\n| GAT | 0.384 | 70.59% | 78.27% |\n| GatedGCN | **0.214** | **76.08%** | **86.51%** |\n| GT | 0.226 | 73.17% | 84.81% |\n\nGT clearly beats GCN and attention-based GAT but remains slightly behind GatedGCN on all three. The authors position it as a stronger attention-based GNN baseline rather than the best model of the time.\n\n## Conclusion\n\nGT's answer to whether graphs should be fully connected is: not yet. Neighbor-only attention works best together with Laplacian positional encoding and BatchNorm, while full connectivity lags badly on all three datasets.\n\nThis conclusion does not last long. Six months later, SAN makes full connectivity work with more careful positional encoding and adjustable weighting for added connections. Another year later, GraphGPS, with Dwivedi also among its authors, puts neighbor-only message passing and all-node attention in the same layer.\n\nBefore using Laplacian eigenvectors as positional encodings, decide how to handle signs. Otherwise the same graph can receive different encodings every time.\n","imageSizes":{"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png":{"width":1324,"height":1000}}}