{"notes":[{"id":"graph-learning/graphormer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graphormer","subtitle":"Better Than You Might Think","author":"Z. Yuan","paperDate":"2021-06-09","publishedOn":"2024-03-28","world":"external/graphormer","paperTitle":"Do Transformers Really Perform Bad for Graph Representation?","paperUrl":"https://arxiv.org/abs/2106.05234","bibliography":{"title":"Do Transformers Really Perform Bad for Graph Representation?","authors":["Chengxuan Ying","Tianle Cai","Shengjie Luo","Shuxin Zheng","Guolin Ke","Di He","Yanming Shen","Tie-Yan Liu"],"url":"https://arxiv.org/abs/2106.05234","pdfUrl":"https://arxiv.org/pdf/2106.05234v5","codeUrl":"https://github.com/Microsoft/Graphormer","preprint":{"id":"2106.05234","submittedOn":"2021-06-09","url":"https://arxiv.org/abs/2106.05234"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html"},"citation":"@misc{graphlearninggraphormer2021,\n  title = {{Do Transformers Really Perform Bad for Graph Representation?}},\n  author = {Chengxuan Ying and Tianle Cai and Shengjie Luo and Shuxin Zheng and Guolin Ke and Di He and Yanming Shen and Tie-Yan Liu},\n  year = {2021},\n  eprint = {2106.05234},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.05234}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphormer.23b94dbe8ef3.bib"},"thumbnail":{"src":"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png","width":872,"height":620},"locale":"en","path":"/content/papers/en/graph-learning/graphormer.438265eee892.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"centrality-encoding","text":"Centrality Encoding","depth":3},{"id":"spatial-encoding","text":"Spatial Encoding","depth":3},{"id":"edge-encoding","text":"Edge Encoding","depth":3},{"id":"other-details","text":"Other Details","depth":3},{"id":"which-gnns-can-it-represent","text":"Which GNNs Can It Represent?","depth":3},{"id":"beyond-wl","text":"Beyond WL","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"pcqm4m-lsc","text":"PCQM4M-LSC","depth":3},{"id":"other-datasets","text":"Other Datasets","depth":3},{"id":"each-encodings-contribution","text":"Each Encoding's Contribution","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""}]},{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"Fully Connected, Worse Results","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"en","path":"/content/papers/en/graph-learning/graph-transformer.c69ec318d0f9.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"attention-only-among-neighbors","text":"Attention Only among Neighbors","depth":3},{"id":"laplacian-positional-encoding","text":"Laplacian Positional Encoding","depth":3},{"id":"edge-features-and-normalization","text":"Edge Features and Normalization","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"what-happens-with-full-connectivity","text":"What Happens with Full Connectivity?","depth":3},{"id":"positional-encoding-and-batchnorm","text":"Positional Encoding and BatchNorm","depth":3},{"id":"comparison-with-gnns","text":"Comparison with GNNs","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"Hearing the Shape of a Graph","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"en","path":"/content/papers/en/graph-learning/san.2fbaa6b9ce67.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"learned-positional-encoding","text":"Learned Positional Encoding","depth":3},{"id":"full-connectivity-two-attention-mechanisms","text":"Full Connectivity, Two Attention Mechanisms","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-large-should-gamma-be","text":"How Large Should \\gamma Be?","depth":3},{"id":"contributions-of-connectivity-and-lpe","text":"Contributions of Connectivity and LPE","depth":3},{"id":"against-the-best-models-of-the-time","text":"Against the Best Models of the Time","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]}],"text":"By 2021, Transformers dominate natural language processing and have found firm footing in computer vision. Graph leaderboards tell a different story.\n\nThe OGB Large-Scale Challenge's PCQM4M-LSC has over 3.8 million molecules, requiring prediction of a quantum-chemistry property from molecular graphs. GT, a graph-adapted Transformer, has validation MAE 0.1400; widening to 83.2 million parameters makes it 0.1408. Neither beats GIN with virtual nodes (GIN-VN), at 0.1395 with only 6.7 million parameters.\n\nChengxuan Ying, Tianle Cai, Shengjie Luo and colleagues at Dalian University of Technology, Princeton, Peking University and Microsoft Research Asia put the question into their title: do Transformers really perform poorly on graphs?\n\nTheir Graphormer uses a standard Transformer and ultimately wins the challenge's graph-level track.\n\n\n> [!TIP]\n> \n> For GT's sparse attention and Laplacian positional encoding, and SAN from the same month, see our notes:\n> \n> - **[[20.12] Graph Transformer: Fully Connected, Worse Results](/blog/papers/graph-learning/graph-transformer)**\n> - **[[21.06] SAN: Hearing the Shape of a Graph](/blog/papers/graph-learning/san)**\n\n\n## Defining the Problem\n\nMany attempts to bring Transformers to graphs, the authors observe, succeed mainly by replacing a GNN component such as aggregation with softmax attention while retaining a GNN backbone.\n\nWhat is missing? Self-attention computes semantic similarity between node features without graph structure. It cannot see which nodes are important, how far apart two nodes lie, or what edge types connect them.\n\nThe key, the authors argue, is to encode that structural information effectively.\n\n## Solving the Problem\n\n[![Figure 1 of the paper: the left shows self-attention. Node features plus centrality encoding undergo Q, K, V projections; QK scores are scaled, biased, softmaxed and multiplied by V. Two central 5×5 grids represent spatial and edge encodings, each assigning a value per node pair and adding it to attention scores. Centrality is added to node features below. The right shows five nodes with size indicating importance and dashed pairwise relations.](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)\n\n*Figure 1 from Ying et al., taken from [**PDF page 3**](https://arxiv.org/pdf/2106.05234v5#page=3), CC BY-SA 4.0. The original caption is omitted.*\n\nGraphormer adds three encodings: one to nodes, two to node pairs, hence attention scores.\n\n### Centrality Encoding\n\nIn social networks, people with more followers tend to be more influential. The authors use simple degree centrality, looking up learnable vectors for each node's in-degree and out-degree and adding them to input features:\n\n$$\nh_i^{(0)}=x_i+z^-_{\\deg^-(v_i)}+z^+_{\\deg^+(v_i)}\n$$\n\nFor undirected graphs, in-degree and out-degree coincide and can be combined. Both Q and K then carry node importance.\n\n### Spatial Encoding\n\nFor sentences, Transformers need explicit word positions. Graphs have no fixed axis, so the authors describe pairwise relationships instead.\n\nThe shortest-path distance $\\phi(v_i,v_j)$ selects a learnable scalar $b_{\\phi(v_i,v_j)}$, added to the attention score:\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}\n$$\n\nDisconnected pairs receive a special distance −1 with its own bias. These biases are shared across all layers.\n\nIf $b$ learns to decrease with distance, nodes favor nearby nodes over distant ones while retaining access to the whole graph.\n\n### Edge Encoding\n\nMolecular bond types are edge features. Earlier approaches added them to endpoint nodes or included them during aggregation, passing edge information only to connected nodes.\n\nGraphormer puts edge features into attention scores too. For each node pair $(v_i,v_j)$, choose a shortest path $(e_1,\\dots,e_N)$, take the inner product of each edge's features with a learnable vector, and average:\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}+c_{ij},\\qquad\nc_{ij}=\\frac1N\\sum_{n=1}^N x_{e_n}\\big(w_n^E\\big)^\\top\n$$\n\n### Other Details\n\nLayerNorm precedes attention and the feedforward network: Pre-LN. A special `[VNode]` connects to all nodes, with its final-layer representation summarizing the graph, like BERT's `[CLS]`. Its distance to every node is 1, but these are not real edges, so its spatial encoding gets a dedicated bias.\n\n### Which GNNs Can It Represent?\n\nWith suitable weights and distance functions, the authors prove that one Graphormer layer represents GIN, GCN or GraphSAGE aggregation and update. For mean aggregation, set distance-1 bias to 0 and others to negative infinity, with $W_Q=W_K=0$ and $W_V=I$. Softmax then averages only neighbors.\n\nVerify on a five-node graph:\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\ndef build(n, edges):\n    M = np.zeros((n, n))\n    for i, j in edges:\n        M[i, j] = M[j, i] = 1\n    return M\n\ndef shortest_paths(M):\n    n = len(M)\n    dist = np.full((n, n), -1)\n    for s in range(n):                       # Breadth-first search from every node\n        dist[s, s], frontier, d = 0, [s], 0\n        while frontier:\n            d += 1\n            nxt = [v for u in frontier for v in range(n) if M[u, v] and dist[s, v] < 0]\n            for v in nxt:\n                dist[s, v] = d\n            frontier = list(dict.fromkeys(nxt))\n    return dist\n\nM = build(5, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)])\nh = rng.normal(size=(5, 3))\nspd = shortest_paths(M)\n\nbias = np.where(spd == 1, 0.0, -np.inf)          # Bias 0 at distance 1, negative infinity elsewhere\nattn = np.exp(bias) / np.exp(bias).sum(1, keepdims=True)   # W_Q = W_K = 0, leaving only the bias\nout = attn @ h                                   # W_V = I\n\nmean_nbrs = np.array([h[M[v] > 0].mean(0) for v in range(5)])\nprint(\"Node 2 attention:\", attn[2].round(2))\nprint(\"Matches neighbor mean:\", np.allclose(out, mean_nbrs))\n```\n\n```text\nNode 2 attention: [0.33 0.33 0.   0.33 0.  ]\nMatches neighbor mean: True\n```\n\nNode 2 has three neighbors, each getting one third; self and distance-2 nodes receive 0. Multiplying the mean by degree gives sum, with degree available through centrality encoding. Max aggregation uses one attention head per dimension to select the maximum; the proof is in the appendix.\n\n### Beyond WL\n\nConversely, Graphormer can see distinctions WL misses. The appendix gives a pair:\n\n[![Appendix Figure 2 of the paper: the left graph is a ladder with two rows of three nodes, four pink corners and two blue middle nodes. The right is two triangles joined by a bridge, with four pink outer nodes and two blue bridge nodes.](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)\n\n*Appendix Figure 2 from Ying et al., taken from [**PDF page 14**](https://arxiv.org/pdf/2106.05234v5#page=14), CC BY-SA 4.0. The original caption is omitted.*\n\nThe left is a ladder, the right two triangles joined by a bridge. Both have four degree-2 nodes and two degree-3 nodes, with the same colors in every WL round. Shortest-path distances differ:\n\n```python\nfrom collections import Counter\n\ndef wl(M, rounds=3):\n    colors = [()] * len(M)                   # All nodes start with one color\n    for _ in range(rounds):                  # New color = self color plus multiset of neighboring colors\n        colors = [(colors[v], tuple(sorted(colors[u] for u in range(len(M)) if M[v, u])))\n                  for v in range(len(M))]\n    return Counter(colors)\n\nladder = build(6, [(0, 1), (1, 2), (3, 4), (4, 5), (0, 3), (1, 4), (2, 5)])\nbridge = build(6, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)])\nprint(\"Same WL color counts:\", wl(ladder) == wl(bridge))\nfor name, G in [(\"Ladder    \", ladder), (\"Two triangles\", bridge)]:\n    print(name, sorted({tuple(sorted(row)) for row in shortest_paths(G).tolist()}))\n```\n\n```text\nSame WL color counts: True\nLadder     [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 2, 3)]\nTwo triangles [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 3, 3)]\n```\n\nEach row gives distances from one node type to all nodes. Blue nodes have (0, 1, 1, 1, 2, 2) in both graphs. A pink ladder corner has only one node at distance 3, while a triangle's outer node has two. Access to distances separates these graphs.\n\nThe authors also prove that self-attention without any encoding can implement a whole-graph mean readout, effectively supplying a virtual node in every layer. Experimentally, increasing depth and width keeps reducing training and validation errors, without observed over-smoothing.\n\n## Discussion\n\n### PCQM4M-LSC\n\nThe main experiments use PCQM4M-LSC. Full Graphormer has 12 layers, hidden dimension 768 and 47.1 million parameters; the smaller version has 6 layers, 512 dimensions and 12.5 million parameters. Training takes around two days on 8 V100s.\n\n| Model | Parameters | Validation MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 2.0M | 0.1691 |\n| GIN | 3.8M | 0.1537 |\n| GIN-VN | 6.7M | 0.1395 |\n| DeeperGCN-VN | 25.5M | 0.1398 |\n| GT | 0.6M | 0.1400 |\n| GT-Wide | 83.2M | 0.1408 |\n| Graphormer (small) | 12.5M | 0.1264 |\n| Graphormer | 47.1M | **0.1234** |\n\nValidation MAE falls 11.5% relative to the previous best GIN-VN. Making GT over a hundred times wider fails to improve it. For the challenge, an ensemble of Graphormer and another model, ExpC, reaches 0.1200 on the full test set, winning the graph-level track.\n\n### Other Datasets\n\nZINC offers the cleanest comparison: all models have around 500,000 parameters and train from scratch. GraphormerSLIM, with 489,000 parameters, achieves test MAE 0.122, versus SAN's 0.139, PNA's 0.142 and GT's 0.226.\n\nThe two OGB molecular datasets require attention to conditions. Graphormer reaches 31.39% average precision on MolPCBA versus the previous best 29.79%, and 80.51% ROC-AUC on MolHIV versus 79.70%. Both versions are pretrained on PCQM4M-LSC, have 119.5 million and 47 million parameters respectively, and use FLAG augmentation against overfitting. For fairness, the authors also report GIN-VN pretrained and fine-tuned in the same way: 29.02% and 77.80%.\n\n### Each Encoding's Contribution\n\nPCQM4M-LSC ablations use 12-layer models trained for only 100,000 steps to save computation:\n\n| Setting | Validation MAE ↓ |\n| --- | ---: |\n| No structural encoding | 0.2276 |\n| Laplacian positional encoding | 0.1483 |\n| Shortest-path spatial encoding | 0.1427 |\n| Spatial + centrality encoding | 0.1396 |\n| Add edge encoding at nodes | 0.1328 |\n| Add edge encoding during aggregation | 0.1327 |\n| Add edge encoding as attention bias | **0.1304** |\n\nWithout structural encoding, Transformer MAE is 0.2276, far worse than encoded versions. Spatial encoding beats Laplacian encoding; edge encoding in attention scores beats the two conventional placements.\n\n### The Cost\n\nThe authors themselves list limitations in the conclusion. Attention is quadratic in nodes, making large graphs unaffordable; node-level tasks need sampling; degrees and shortest paths are general choices, but particular domains could use more suitable encodings, such as 3D distances between atoms.\n\n## Conclusion\n\nGraphormer's answer to its title is that Transformers can perform well on graphs, provided structure is supplied. Add degrees to nodes and shortest-path distances and path edges to attention scores, and a standard Transformer beats all GNNs on the largest graph-level dataset of the time.\n\nA year later, GraphGPS pushes PCQM4Mv2 validation MAE lower: GPS-medium with 19.4 million parameters reaches 0.0858, versus Graphormer with 48.3 million at 0.0864—almost tied, with less than half the parameters.\n\nFor Transformers on small molecules, add structural encodings first. At the same 100,000 training steps, removing them worsens MAE from 0.13 to 0.23.\n","imageSizes":{"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png":{"width":872,"height":620},"/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png":{"width":908,"height":216}}}