{"notes":[{"id":"graph-learning/graphormer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graphormer","subtitle":"其实不差","author":"Z. Yuan","paperDate":"2021-06-09","publishedOn":"2024-03-28","world":"external/graphormer","paperTitle":"Do Transformers Really Perform Bad for Graph Representation?","paperUrl":"https://arxiv.org/abs/2106.05234","bibliography":{"title":"Do Transformers Really Perform Bad for Graph Representation?","authors":["Chengxuan Ying","Tianle Cai","Shengjie Luo","Shuxin Zheng","Guolin Ke","Di He","Yanming Shen","Tie-Yan Liu"],"url":"https://arxiv.org/abs/2106.05234","pdfUrl":"https://arxiv.org/pdf/2106.05234v5","codeUrl":"https://github.com/Microsoft/Graphormer","preprint":{"id":"2106.05234","submittedOn":"2021-06-09","url":"https://arxiv.org/abs/2106.05234"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html"},"citation":"@misc{graphlearninggraphormer2021,\n  title = {{Do Transformers Really Perform Bad for Graph Representation?}},\n  author = {Chengxuan Ying and Tianle Cai and Shengjie Luo and Shuxin Zheng and Guolin Ke and Di He and Yanming Shen and Tie-Yan Liu},\n  year = {2021},\n  eprint = {2106.05234},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.05234}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphormer.23b94dbe8ef3.bib"},"thumbnail":{"src":"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png","width":872,"height":620},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphormer.3a73afc450c3.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"中心性编码","text":"中心性编码","depth":3},{"id":"空间编码","text":"空间编码","depth":3},{"id":"边编码","text":"边编码","depth":3},{"id":"其他细节","text":"其他细节","depth":3},{"id":"能表示哪些-gnn","text":"能表示哪些 GNN？","depth":3},{"id":"比-wl-更强","text":"比 WL 更强","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"pcqm4m-lsc","text":"PCQM4M-LSC","depth":3},{"id":"其他数据集","text":"其他数据集","depth":3},{"id":"每个编码贡献多少","text":"每个编码贡献多少","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""}]},{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"连满反而糟","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graph-transformer.1ebf78897bd9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"只在邻居之间算注意力","text":"只在邻居之间算注意力","depth":3},{"id":"拉普拉斯位置编码","text":"拉普拉斯位置编码","depth":3},{"id":"边特征与归一化","text":"边特征与归一化","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"全连接会怎样","text":"全连接会怎样？","depth":3},{"id":"位置编码与-batchnorm","text":"位置编码与 BatchNorm","depth":3},{"id":"和-gnn-比","text":"和 GNN 比","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"听见图的形状","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/san.2705d642b9c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"学出来的位置编码","text":"学出来的位置编码","depth":3},{"id":"全连接但分开两套注意力","text":"全连接，但分开两套注意力","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"gamma-该设多大","text":"\\gamma 该设多大？","depth":3},{"id":"全连接和-lpe-各自的贡献","text":"全连接和 LPE 各自的贡献","depth":3},{"id":"和当时最好的模型比","text":"和当时最好的模型比","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]}],"text":"2021 年，Transformer 已经是自然语言处理的主流，在计算机视觉也站稳了脚步。图的排行榜却是另一回事。\n\nOGB 大规模挑战赛的 PCQM4M-LSC 有 380 多万个分子，要从分子图预测一个量子化学性质。这个数据集上，把 Transformer 改写到图上的 GT，验证集 MAE 是 0.1400；加宽到 8,320 万参数，反而是 0.1408。两者都没赢过只有 670 万参数、加了虚拟节点的 GIN（GIN-VN，0.1395）。\n\n大连理工大学、普林斯顿大学、北京大学与微软亚洲研究院的 Chengxuan Ying、Tianle Cai、Shengjie Luo 等人，干脆把疑问写进论文标题，问 Transformer 在图上是不是真的表现很差。\n\n他们的 Graphormer 用的是标准的 Transformer，最后拿下这场比赛图层级赛道的第一名。\n\n\n> [!TIP]\n> \n> GT 的稀疏注意力与拉普拉斯位置编码，以及同一个月发表的 SAN，可以参考我们的笔记：\n> \n> - **[[20.12] Graph Transformer：连满反而糟](/blog/papers/graph-learning/graph-transformer)**\n> - **[[21.06] SAN：听见图的形状](/blog/papers/graph-learning/san)**\n\n\n## 定义问题\n\n作者观察到，把 Transformer 用在图上的尝试不少，真正有效的多半是把 GNN 里的某个模块（例如聚合）换成 softmax 注意力，骨架还是 GNN。\n\n问题出在哪里？自注意力计算的是两个节点特征之间的语义相似度，图的结构完全没进入这个计算。哪个节点比较重要、两个节点在图上隔多远、中间的边是什么类型，注意力都看不到。\n\n作者认为，关键在于把这些结构信息有效地编码进模型。\n\n## 解决问题\n\n[![原论文 Figure 1：左边是一层自注意力，节点特征加上中心性编码后，经 Q、K、V 的线性投影，Q 和 K 相乘、缩放，再加上一个偏置后做 softmax，乘上 V。中间两个 5×5 的方格分别是空间编码与边编码，各自给每一对节点一个值，两者都加到注意力分数上；下方的中心性编码与节点特征相加。右边是一张五个节点的小图，节点大小代表重要性，虚线代表节点之间的关系。](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)\n\n*原论文 Figure 1（Ying 等人），取自 [**PDF 第 3 页**](https://arxiv.org/pdf/2106.05234v5#page=3)，CC BY-SA 4.0。图注未收录。*\n\nGraphormer 加了三种编码。一种加在节点上，另外两种加在节点与节点之间，也就是注意力分数上。\n\n### 中心性编码\n\n社交网络里，关注者多的人通常更有影响力。作者用最简单的度数中心性，依每个节点的入度和出度各查一个可学的向量，加到输入的节点特征上：\n\n$$\nh_i^{(0)}=x_i+z^-_{\\deg^-(v_i)}+z^+_{\\deg^+(v_i)}\n$$\n\n无向图的入度和出度相同，合成一个就好。这样 Q 和 K 里都带着节点的重要性。\n\n### 空间编码\n\nTransformer 处理句子时，要另外告诉模型每个字的位置。图没有一条固定的轴，作者改成描述两个节点之间的关系。\n\n他们用两个节点之间的最短路径距离 $\\phi(v_i,v_j)$，每一种距离对应一个可学的标量 $b_{\\phi(v_i,v_j)}$，加在注意力分数上：\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}\n$$\n\n两个节点不连通时，距离设成一个特殊值 −1，也有自己的偏置。这组偏置在所有层之间共用。\n\n如果 $b$ 学成随距离递减，每个节点就会多看近的、少看远的；但它仍然看得到全图。\n\n### 边编码\n\n分子的化学键类型是边的特征。以前的做法是把边特征加到两端节点上，或在聚合时一起用，边的信息只传到相连的节点。\n\nGraphormer 把边特征也放进注意力分数。对每一对节点 $(v_i,v_j)$，找一条最短路径 $(e_1,\\dots,e_N)$，把路上每条边的特征和一个可学的向量做内积，取平均：\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}+c_{ij},\\qquad\nc_{ij}=\\frac1N\\sum_{n=1}^N x_{e_n}\\big(w_n^E\\big)^\\top\n$$\n\n### 其他细节\n\nLayerNorm 放在注意力和前馈网络之前，也就是 Pre-LN。另外加一个和所有节点相连的特殊节点 `[VNode]`，最后一层它的表示就是整张图的表示，作用类似 BERT 的 `[CLS]`。它和每个节点的距离都是 1，但这条连接不是真的边，所以它的空间编码另外用一个专属的偏置。\n\n### 能表示哪些 GNN？\n\n作者证明，选对权重和距离函数，Graphormer 的一层可以表示 GIN、GCN、GraphSAGE 的聚合与更新。以平均聚合为例，把距离 1 的偏置设成 0、其他设成负无穷大，再让 $W_Q=W_K=0$、$W_V=I$，softmax 就只会平均分给邻居。\n\n拿一张五个节点的小图验证一次：\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\ndef build(n, edges):\n    M = np.zeros((n, n))\n    for i, j in edges:\n        M[i, j] = M[j, i] = 1\n    return M\n\ndef shortest_paths(M):\n    n = len(M)\n    dist = np.full((n, n), -1)\n    for s in range(n):                       # 从每个节点做一次广度优先搜索\n        dist[s, s], frontier, d = 0, [s], 0\n        while frontier:\n            d += 1\n            nxt = [v for u in frontier for v in range(n) if M[u, v] and dist[s, v] < 0]\n            for v in nxt:\n                dist[s, v] = d\n            frontier = list(dict.fromkeys(nxt))\n    return dist\n\nM = build(5, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)])\nh = rng.normal(size=(5, 3))\nspd = shortest_paths(M)\n\nbias = np.where(spd == 1, 0.0, -np.inf)          # 距离 1 给 0，其他给负无穷大\nattn = np.exp(bias) / np.exp(bias).sum(1, keepdims=True)   # W_Q = W_K = 0，只剩偏置\nout = attn @ h                                   # W_V = I\n\nmean_nbrs = np.array([h[M[v] > 0].mean(0) for v in range(5)])\nprint(\"节点 2 的注意力:\", attn[2].round(2))\nprint(\"和邻居平均相同:\", np.allclose(out, mean_nbrs))\n```\n\n```text\n节点 2 的注意力: [0.33 0.33 0.   0.33 0.  ]\n和邻居平均相同: True\n```\n\n节点 2 有三个邻居，各分到三分之一，自己和距离 2 的节点分到 0。平均乘上度数就是加总，度数可以从中心性编码取得；最大值聚合则要每个维度用一个注意力头去挑，证明在论文附录。\n\n### 比 WL 更强\n\n反过来，Graphormer 也看得到 WL 看不到的东西。论文附录举了一对图：\n\n[![原论文附录 Figure 2：左边是排成两行、每行三个节点的梯子形，四个角落的节点是粉红色，中间两个是蓝色；右边是两个三角形用一座桥相连，外侧四个节点是粉红色，桥上两个是蓝色。](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)\n\n*原论文附录 Figure 2（Ying 等人），取自 [**PDF 第 14 页**](https://arxiv.org/pdf/2106.05234v5#page=14)，CC BY-SA 4.0。图注未收录。*\n\n左边是梯子形，右边是两个三角形用一座桥相连。两张图都有四个度数 2 的节点、两个度数 3 的节点，WL 每一轮涂出的颜色也一样。看最短路径距离就不同了：\n\n```python\nfrom collections import Counter\n\ndef wl(M, rounds=3):\n    colors = [()] * len(M)                   # 一开始大家同色\n    for _ in range(rounds):                  # 新颜色 = 自己的颜色 + 邻居颜色的多重集合\n        colors = [(colors[v], tuple(sorted(colors[u] for u in range(len(M)) if M[v, u])))\n                  for v in range(len(M))]\n    return Counter(colors)\n\nladder = build(6, [(0, 1), (1, 2), (3, 4), (4, 5), (0, 3), (1, 4), (2, 5)])\nbridge = build(6, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)])\nprint(\"WL 的颜色统计相同:\", wl(ladder) == wl(bridge))\nfor name, G in [(\"梯子形    \", ladder), (\"两个三角形\", bridge)]:\n    print(name, sorted({tuple(sorted(row)) for row in shortest_paths(G).tolist()}))\n```\n\n```text\nWL 的颜色统计相同: True\n梯子形     [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 2, 3)]\n两个三角形 [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 3, 3)]\n```\n\n每一行是某一类节点到其他节点的距离。蓝色节点两边都是 (0, 1, 1, 1, 2, 2)；粉红色的节点，梯子形的角落最远只到 3 一次，三角形外侧的节点有两个距离 3 的邻居。只要模型看得到距离，这两张图就分得开。\n\n作者还证明了另一件事。没有任何编码时，自注意力本身就能表示对全图取平均的读出，等于每一层都附带一个虚拟节点。实验中他们也观察到，模型加深、加宽之后，训练和验证误差都持续下降，没有出现 over-smoothing。\n\n## 讨论\n\n### PCQM4M-LSC\n\n主要的实验在 PCQM4M-LSC 上。Graphormer 有两个大小，12 层、隐藏维度 768 的完整版（4,710 万参数），以及 6 层、512 维的小版（1,250 万参数），在 8 张 V100 上训练约两天。\n\n| 模型 | 参数 | 验证 MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 2.0M | 0.1691 |\n| GIN | 3.8M | 0.1537 |\n| GIN-VN | 6.7M | 0.1395 |\n| DeeperGCN-VN | 25.5M | 0.1398 |\n| GT | 0.6M | 0.1400 |\n| GT-Wide | 83.2M | 0.1408 |\n| Graphormer（小） | 12.5M | 0.1264 |\n| Graphormer | 47.1M | **0.1234** |\n\n相对于先前最好的 GIN-VN，验证 MAE 降了 11.5%。GT 加宽了一百多倍，表现却没有变好。比赛提交时，作者把 Graphormer 和另一个模型 ExpC 集成，在完整测试集上做到 0.1200，拿下图层级赛道的第一名。\n\n### 其他数据集\n\nZINC 是最干净的比较。所有模型都控制在 50 万参数左右、从头训练，48.9 万参数的 GraphormerSLIM 测试 MAE 是 0.122，SAN 是 0.139，PNA 是 0.142，GT 是 0.226。\n\nOGB 的两个分子数据集要看条件。MolPCBA 上 Graphormer 的平均精确率是 31.39%，先前最好的是 29.79%；MolHIV 上 ROC-AUC 是 80.51%，先前最好的是 79.70%。这两个结果的 Graphormer 先在 PCQM4M-LSC 上预训练，参数分别是 1.195 亿和 4,700 万，还用了 FLAG 数据增强对抗过拟合。为了公平，作者也列了同样在 PCQM4M-LSC 上预训练、再微调的 GIN-VN，分别是 29.02% 和 77.80%。\n\n### 每个编码贡献多少\n\n作者在 PCQM4M-LSC 上做消融，为了省计算，模型都是 12 层、只训练 10 万步：\n\n| 设置 | 验证 MAE ↓ |\n| --- | ---: |\n| 不加任何结构编码 | 0.2276 |\n| 拉普拉斯位置编码 | 0.1483 |\n| 最短路径的空间编码 | 0.1427 |\n| 空间编码＋中心性编码 | 0.1396 |\n| 再加上边编码（加在节点上） | 0.1328 |\n| 再加上边编码（聚合时使用） | 0.1327 |\n| 再加上边编码（注意力偏置） | **0.1304** |\n\n没有任何结构编码的 Transformer，MAE 是 0.2276，和加上编码的版本差了一大截。空间编码胜过拉普拉斯位置编码；边编码放进注意力分数，比传统的两种放法都好。\n\n### 代价\n\n结论里，作者自己点出几个限制。注意力的计算量是节点数的平方，大图用不起；节点层级的任务需要一套采样的方法；度数和最短路径只是通用的选择，特定领域可以换成更合适的编码，例如分子里两个原子的 3D 距离。\n\n## 结论\n\nGraphormer 对标题的回答是，Transformer 在图上其实不差，前提是把结构放进去。度数加在节点上，最短路径距离和路上的边加在注意力分数上，一个标准的 Transformer 就在当时最大的图层级数据集上赢过所有 GNN。\n\n一年后的 GraphGPS 把 PCQM4Mv2 的验证 MAE 再往下推，1,940 万参数的 GPS-medium 是 0.0858，4,830 万参数的 Graphormer 是 0.0864，两者几乎打平，参数却少了一半多。\n\n如果你要在小分子上用 Transformer，先把结构编码加上去再谈别的。同样训练 10 万步，拿掉它们，MAE 就从 0.13 退到 0.23。\n","imageSizes":{"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png":{"width":872,"height":620},"/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png":{"width":908,"height":216}}}