{"notes":[{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"连满反而糟","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graph-transformer.1ebf78897bd9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"只在邻居之间算注意力","text":"只在邻居之间算注意力","depth":3},{"id":"拉普拉斯位置编码","text":"拉普拉斯位置编码","depth":3},{"id":"边特征与归一化","text":"边特征与归一化","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"全连接会怎样","text":"全连接会怎样？","depth":3},{"id":"位置编码与-batchnorm","text":"位置编码与 BatchNorm","depth":3},{"id":"和-gnn-比","text":"和 GNN 比","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"听见图的形状","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/san.2705d642b9c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"学出来的位置编码","text":"学出来的位置编码","depth":3},{"id":"全连接但分开两套注意力","text":"全连接，但分开两套注意力","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"gamma-该设多大","text":"\\gamma 该设多大？","depth":3},{"id":"全连接和-lpe-各自的贡献","text":"全连接和 LPE 各自的贡献","depth":3},{"id":"和当时最好的模型比","text":"和当时最好的模型比","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"图上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphgps.74aedb083e17.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"位置与结构","text":"位置与结构","depth":3},{"id":"为什么消息传递需要它们","text":"为什么消息传递需要它们","depth":3},{"id":"先传递再注意会先丢掉信息","text":"先传递再注意，会先丢掉信息","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一层-gps","text":"一层 GPS","depth":3},{"id":"线性的复杂度","text":"线性的复杂度","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"消融两条路各自的份量","text":"消融：两条路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年后的重新评估","text":"一年后的重新评估","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"Transformer 处理一句话时，每个字都会看其他所有的字。换个角度看，这就是把一句话当成一张全连接的图，每个字是一个节点，任两个字之间都有一条边。\n\n既然句子已经是图，真正的图能不能直接丢进 Transformer？\n\n南洋理工大学的 Vijay Prakash Dwivedi 与 Xavier Bresson 认为不能照搬。句子没有现成的连接关系，通常也只有几十、几百个字，全连接合理，也算得动；图却自带结构，节点还可能多到上百万个，直接连满既浪费了这份结构，也算不起。\n\n他们在 2020 年底提出的 Graph Transformer（下文简称 GT），是把 Transformer 改写到任意图上的早期尝试，半年后的 SAN、再隔一年的 GraphGPS 都拿它当比较的起点。\n\n\n> [!TIP]\n> \n> 后来让全连接重新站起来的 SAN，以及把两种做法放进同一层的 GraphGPS，可以参考我们的笔记：\n> \n> - **[[21.06] SAN：听见图的形状](/blog/papers/graph-learning/san)**\n> - **[[22.05] GraphGPS：图上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n## 定义问题\n\n作者把问题拆成两件事。\n\n**稀疏性**。更早的图 Transformer（Li 等人 2019）让每个节点看所有节点，以取得全局信息。作者认为这样丢掉了图的稀疏连接，而稀疏连接本身就是很好的归纳偏置；全局信息可以改用位置编码、虚拟节点等方法补。\n\n**位置编码**。句子里的字有先后，Transformer 用正弦波替每个位置编码。图没有标准的节点顺序，还有对称性，很难定义一个唯一的位置。作者认为，只在邻居之间算注意力的 GAT 表现不够好，原因之一就是缺了位置信息。\n\n这篇要验证的是，只看邻居的注意力，加上合适的位置编码，能不能做出一个通用又好用的图 Transformer。\n\n## 解决问题\n\n[![原论文 Figure 1：两张方块图。左边是 Graph Transformer 层，节点特征加上拉普拉斯位置编码 λ 后，经过 Q、K、V 投影、相乘、缩放、softmax 与加总，再接 Add&Norm、FFN、Add&Norm，重复 L 层；右边是带边特征的版本，多了一条边特征 e_ij 的路径，注意力分数和 E 投影后的边特征相乘，边的表示也有自己的 FFN 与归一化。](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)\n\n*原论文 Figure 1（Dwivedi 与 Bresson），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2012.09699v2#page=2)。图注未收录。*\n\n左边是基本版，右边是加上边特征的版本。和原本的 Transformer 相比，改动有三处。\n\n### 只在邻居之间算注意力\n\n一层 GT 的注意力和 Transformer 几乎一样，差别只在加总的范围：\n\n$$\n\\hat h_i^{\\ell+1}=O_h^\\ell\\,\\Big\\Vert_{k=1}^{H}\\Big(\\sum_{j\\in\\mathcal N_i}w_{ij}^{k,\\ell}V^{k,\\ell}h_j^\\ell\\Big),\\qquad\nw_{ij}^{k,\\ell}=\\mathrm{softmax}_j\\Big(\\frac{Q^{k,\\ell}h_i^\\ell\\cdot K^{k,\\ell}h_j^\\ell}{\\sqrt{d_k}}\\Big)\n$$\n\n$\\mathcal N_i$ 是节点 $i$ 的邻居，$\\Vert$ 表示把 $H$ 个注意力头的结果接起来。Transformer 对所有位置加总，GT 只对邻居加总。之后照样接残差、归一化和前馈网络。\n\n### 拉普拉斯位置编码\n\n位置编码用的是拉普拉斯矩阵的特征向量：\n\n$$\n\\Delta=I-D^{-1/2}AD^{-1/2}=U^\\top\\Lambda U\n$$\n\n作者取最小的 $k$ 个非平凡特征向量，也就是跳过特征值 0 的那一条，每个节点在这 $k$ 条向量上的值，就是它的位置编码 $\\lambda_i$。这组数字经过一层线性投影，只在输入层加到节点特征上，中间的层不再加。\n\n为什么是特征向量？作者的理由是，它们是 NLP 正弦位置编码在图上的推广，能把距离的信息带进来，相近的节点拿到相近的编码。\n\n拿一张两个三角形用一座桥相连的小图试算。A、B、C 在左边，D、E、F 在右边，C 和 D 是桥，取 $k=2$：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\nnames = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)]\nA = np.zeros((6, 6))\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\n\nd = A.sum(1)\nL = np.eye(6) - A / np.sqrt(np.outer(d, d))      # Δ = I - D^-1/2 A D^-1/2\nlam, U = np.linalg.eigh(L)                      # 特征值由小到大\nU = U * np.sign(U[0])                           # 我们自己定的规矩：让 A 那一格为正\n\npe = U[:, 1:3]                                  # 跳过特征值 0，取接下来的 2 条\nprint(\"特征值:\", lam[1:3])\nfor name, row in zip(names, pe):\n    print(name, row)\n```\n\n```text\n特征值: [0.2  1.17]\nA [0.45 0.33]\nB [0.45 0.33]\nC [ 0.32 -0.53]\nD [-0.32 -0.53]\nE [-0.45  0.33]\nF [-0.45  0.33]\n```\n\n第一维把左右两群分开，左边三角形为正、右边为负；第二维把桥上的 C、D 和外侧的四个节点分开。A 和 B 拿到一模一样的编码，因为它们在图上本来就对称，只看结构的编码分不出它们。\n\n特征向量还有一个麻烦，正负号是任意的。上面那行 `np.sign` 是我们自己定的规矩；把整条向量乘上 −1，它照样是特征向量：\n\n```python\nv = pe[:, 0]\nprint(\"原本的向量:\", np.allclose(L @ v, lam[1] * v))\nprint(\"翻号之后  :\", np.allclose(L @ -v, lam[1] * -v))\n```\n\n```text\n原本的向量: True\n翻号之后  : True\n```\n\n同一张图，可能拿到两种相反的位置编码；取 $k$ 条向量，就有 $2^k$ 种组合。换一张图，「让第一个节点为正」这种规矩也未必说得出道理。GT 的做法是训练时随机翻转正负号，让模型学着不在乎。\n\n### 边特征与归一化\n\n化学分子的键类型、知识图谱里的关系，这些信息放在边上。GT 的边特征版本先算出邻居之间的注意力分数，再和投影后的边特征 $E^{k,\\ell}e_{ij}^\\ell$ 相乘，然后才做 softmax；边的表示也有自己的前馈网络，一层一层往上更新。\n\n最后一个改动是归一化。作者把 Transformer 的 LayerNorm 换成 BatchNorm，发现训练更快，泛化也更好。\n\n## 讨论\n\n### 实验设置\n\n作者用 Benchmarking GNNs 的三个数据集：\n\n- **ZINC**：分子的图回归，取 1.2 万个分子的子集，预测溶解度相关的性质，指标是 MAE，越低越好。键的类型是边特征，所以用带边特征的 GT。\n- **PATTERN**：用随机区块模型生成的 1.4 万张图，每个节点分成两类。\n- **CLUSTER**：同样是生成的 1.2 万张图，每个节点要分到 6 个群之一。\n\n模型都是 10 层、8 个注意力头，参数控制在 50 万左右，每组实验跑 4 个随机数种子。\n\n### 全连接会怎样？\n\n作者把同一个模型分别放在原本的图，以及把所有节点两两连起来的全连接图上。用 BatchNorm 和拉普拉斯位置编码时：\n\n| 数据集 | 只看邻居 | 看所有节点 |\n| --- | ---: | ---: |\n| ZINC（MAE ↓） | 0.226 | 0.598 |\n| CLUSTER（准确率 ↑） | 73.17% | 27.12% |\n| PATTERN（准确率 ↑） | 84.81% | 54.94% |\n\n三个数据集都大幅变差。PATTERN 只有两类，54.94% 离乱猜不远；CLUSTER 有 6 类，27.12% 也只比乱猜的 16.7% 好一些。ZINC 的全连接版本没有用边特征，因为作者要看的就是完全没有稀疏结构的情况。\n\n作者的结论是，在结构任意的图上，稀疏连接是关键的归纳偏置。\n\n### 位置编码与 BatchNorm\n\n作者比较了 LayerNorm／BatchNorm 与有无位置编码的四种组合，三个数据集上最好的都是 BatchNorm 加拉普拉斯位置编码。有些组合的变异大得吓人，例如 PATTERN 上用 LayerNorm、不加位置编码，四个种子的准确率是 73.14% ± 13.63%。\n\n作者也拿 Graph-BERT 用的 WL 位置编码来比：\n\n| 位置编码 | ZINC（MAE ↓） | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| 不用 | 0.264 | 72.14% | 83.95% |\n| 拉普拉斯 | **0.226** | **73.17%** | **84.81%** |\n| WL（Graph-BERT） | 0.267 | 70.79% | 75.49% |\n\nWL 位置编码在 PATTERN 上的训练准确率是 97.03%，测试只有 75.49%，作者认为它在这类生成的图上容易过拟合。\n\n### 和 GNN 比\n\n最后和同样 50 万参数的 GNN 比：\n\n| 模型 | ZINC（MAE ↓） | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| GCN | 0.367 | 68.50% | 71.89% |\n| GAT | 0.384 | 70.59% | 78.27% |\n| GatedGCN | **0.214** | **76.08%** | **86.51%** |\n| GT | 0.226 | 73.17% | 84.81% |\n\nGT 明显赢过 GCN 和同样用注意力的 GAT，但三个数据集都还差 GatedGCN 一点。作者把 GT 定位成一个新的、更强的注意力式 GNN 基准，而不是当时最好的模型。\n\n## 结论\n\nGT 对「图要不要连满」的回答是先别连。只看邻居的注意力、拉普拉斯位置编码和 BatchNorm 一起用时效果最好，全连接的版本在三个数据集上都大幅落后。\n\n这个结论没有维持太久。半年后的 SAN 让全连接重新站起来，前提是位置编码做得更讲究，而且让模型自己调配补上的连接分到多少比重；再隔一年，Dwivedi 自己也是作者之一的 GraphGPS，干脆把只看邻居的消息传递和看所有节点的注意力放进同一层。\n\n拿拉普拉斯特征向量当位置编码之前，先想好正负号怎么处理。不然同一张图，每次算出来的编码都可能不一样。\n","imageSizes":{"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png":{"width":1324,"height":1000}}}