{"notes":[{"id":"graph-learning/mpnn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"MPNN","subtitle":"原来都在传话","author":"Z. Yuan","paperDate":"2017-04-04","publishedOn":"2023-10-11","world":"external/mpnn","paperTitle":"Neural Message Passing for Quantum Chemistry","paperUrl":"https://arxiv.org/abs/1704.01212","bibliography":{"title":"Neural Message Passing for Quantum Chemistry","authors":["Justin Gilmer","Samuel S. Schoenholz","Patrick F. Riley","Oriol Vinyals","George E. Dahl"],"url":"https://arxiv.org/abs/1704.01212","pdfUrl":"https://arxiv.org/pdf/1704.01212v2","codeUrl":null,"preprint":{"id":"1704.01212","submittedOn":"2017-04-04","url":"https://arxiv.org/abs/1704.01212"},"publication":{"venue":"ICML","year":2017,"kind":"conference","url":"https://proceedings.mlr.press/v70/gilmer17a.html"},"citation":"@misc{graphlearningmpnn2017,\n  title = {{Neural Message Passing for Quantum Chemistry}},\n  author = {Justin Gilmer and Samuel S. Schoenholz and Patrick F. Riley and Oriol Vinyals and George E. Dahl},\n  year = {2017},\n  eprint = {1704.01212},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.01212}\n}\n","citationPath":"/content/papers/citations/graph-learning/mpnn.5c9aab7ce227.bib"},"thumbnail":{"src":"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png","width":628,"height":388},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/mpnn.6c337c26e992.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"消息传递的框架","text":"消息传递的框架","depth":3},{"id":"作者试了哪些变化","text":"作者试了哪些变化","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"qm9-的主要结果","text":"QM9 的主要结果","depth":3},{"id":"没有空间信息时","text":"没有空间信息时","depth":3},{"id":"虚拟边与主节点","text":"虚拟边与主节点","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/over-squashing.1bfa8bae9091.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"问题半径","text":"问题半径","depth":3},{"id":"和-rnn-的瓶颈比较","text":"和 RNN 的瓶颈比较","depth":3},{"id":"只看传播比例","text":"只看传播比例","depth":3},{"id":"一个刻意设计的任务","text":"一个刻意设计的任务","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"全连接的最后一层","text":"全连接的最后一层","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物数据集","text":"生物数据集","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"图上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphgps.74aedb083e17.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"位置与结构","text":"位置与结构","depth":3},{"id":"为什么消息传递需要它们","text":"为什么消息传递需要它们","depth":3},{"id":"先传递再注意会先丢掉信息","text":"先传递再注意，会先丢掉信息","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一层-gps","text":"一层 GPS","depth":3},{"id":"线性的复杂度","text":"线性的复杂度","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"消融两条路各自的份量","text":"消融：两条路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年后的重新评估","text":"一年后的重新评估","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"要知道一个分子的偶极矩、能隙或零点振动能，化学家通常得跑一次密度泛函理论（DFT）的计算。一个小分子大约要一千秒。\n\n如果药物或材料的筛选要看上百万个候选分子，这个速度就成了瓶颈。\n\nGoogle Brain、Google 与 DeepMind 的 Justin Gilmer、Samuel Schoenholz、Patrick Riley、Oriol Vinyals 与 George Dahl 想用神经网络直接从分子图预测这些性质，一个分子只要百分之一秒左右。\n\n当时已经有不少能吃分子图的神经网络，各自取了不同的名字。这篇论文先做的事，是把它们放进同一个框架里。\n\n[![原论文 Figure 1：一个有机分子有两条路通往性质预测。上面的路是 DFT 计算，约需 10 的 3 次方秒；下面的路是消息传递神经网络，原子之间一步步交换消息，约需 10 的负 2 次方秒，两条路的输出都是能量、振动频率等目标值。](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)\n\n*原论文 Figure 1（Gilmer 等人），取自 [**PDF 第 1 页**](https://arxiv.org/pdf/1704.01212v2#page=1)。图注未收录。*\n\n## 定义问题\n\n作者指出，机器学习在化学上的应用，多数还停在特征工程。研究者先手工设计分子的表示法，例如 Coulomb 矩阵、化学键袋（Bag of Bonds）、分子指纹，再接一个现成的回归模型。\n\n神经网络已经有人试过，但还没被广泛采用。作者把这比作卷积网络普及以前的图像模型，缺的是证据，证明带有适当归纳偏置的神经网络，在这个领域真的有用。\n\n分子有天然的对称性。把原子重新编号，分子还是同一个分子；适合它的模型，对图的同构应该不变。作者想做两件事，一是整理出现有图神经网络的共同形式，二是把这个形式在一个实际的基准上推到极限。\n\n基准是 QM9。约 13 万个小分子，每个分子有 13 种用 DFT 算出的性质。评估时用一个很直观的标准，化学界订的「化学精度」；误差低于它，就等于达到化学家能接受的准确度。\n\n## 解决问题\n\n### 消息传递的框架\n\nMPNN（Message Passing Neural Networks）把前向传递分成两个阶段。\n\n**消息传递阶段**跑 $T$ 步。每一步，节点 $v$ 从每个邻居 $w$ 收到一则消息，加总之后更新自己的状态：\n\n$$\nm_v^{t+1}=\\sum_{w\\in N(v)}M_t\\left(h_v^t,h_w^t,e_{vw}\\right)\n$$\n\n$$\nh_v^{t+1}=U_t\\left(h_v^t,m_v^{t+1}\\right)\n$$\n\n$M_t$ 是消息函数，看得到收发双方的状态和边的特征 $e_{vw}$；$U_t$ 是更新函数。\n\n**读出阶段**把所有节点的最终状态合成整张图的表示：\n\n$$\n\\hat y=R\\left(\\left\\{h_v^T\\mid v\\in G\\right\\}\\right)\n$$\n\n$R$ 必须对节点的排列不变。三个函数都是可学习的。\n\n作者接着逐一说明，至少八个既有的模型都能写成这个形式，包括 Duvenaud 等人的分子指纹卷积、门控图神经网络（GG-NN）、Interaction Networks、分子图卷积、深度张量网络，以及三个以拉普拉斯为基础的频域方法（Bruna 等人、ChebNet、GCN）。\n\n以 GCN 为例，消息函数是「邻居的状态乘上权重，再依两端的度数缩放」，更新函数是 ReLU。拿一张五个节点的小图，用消息与更新函数的写法算一次，再和 GCN 的矩阵写法对照：\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\nn = 5\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]\nnbrs = {v: [v] for v in range(n)}            # 每个节点先放自己：自环也算一条边\nfor i, j in edges:\n    nbrs[i].append(j)\n    nbrs[j].append(i)\ndeg = {v: len(nbrs[v]) for v in range(n)}    # 加上自环后的度数\nh = rng.normal(size=(n, 3))\nW = rng.normal(size=(3, 3))\n\ndef message(h_v, h_w, v, w):                 # M_t(h_v, h_w, e_vw)\n    return h_w @ W / np.sqrt(deg[v] * deg[w])\n\ndef update(h_v, m_v):                        # U_t(h_v, m_v)\n    return np.maximum(m_v, 0)\n\ndef mpnn_step(h):\n    return np.array([update(h[v], sum(message(h[v], h[w], v, w) for w in nbrs[v]))\n                     for v in range(n)])\n\n# 对照 GCN 的矩阵写法 ReLU(Â H W)\nA = np.eye(n)\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nprint(\"和 GCN 的矩阵写法相同:\", np.allclose(mpnn_step(h), np.maximum(A_hat @ h @ W, 0)))\n```\n\n```text\n和 GCN 的矩阵写法相同: True\n```\n\n这两种写法算的是同一件事，只是一个从单一节点的角度描述，一个把整张图写成矩阵。框架的好处在于，换一个 `message` 或 `update`，就是另一个模型。\n\n### 作者试了哪些变化\n\n有了共同的形式，作者在每个组件上各试几种选择：\n\n- **消息函数**：GG-NN 的作法是依边的类型挑一个矩阵乘上邻居状态。作者提出 **edge network**，用一个神经网络把连续的边特征向量映射成 $d\\times d$ 的矩阵，这样原子间距离这类连续值也能放进来。另一种 **pair message** 让消息同时看收发双方的状态，但训练比较不稳定，作者没有继续深入。\n- **虚拟的图元素**：一种是在没有相连的节点对之间，加上一种特殊类型的「虚拟边」；另一种是加一个和所有节点相连的**主节点**，当作每一步大家都能读写的全局暂存区。两者都让信息在传递阶段走得更远。\n- **读出函数**：除了 GG-NN 的加总式读出，作者也试了为集合设计的 set2set，表达力应该更强。\n- **多塔（towers）**：把 $d$ 维的节点状态切成 $k$ 份，各自做消息传递，再用一个共享的网络混合。同样的计算量下，可以用更大的隐藏状态。\n\n输入方面，作者发现把氢原子当成明确的节点、放进完整的边特征（键的类型和原子间距离），对很多目标都很重要。\n\n## 讨论\n\n### QM9 的主要结果\n\n评估指标是**误差比**，模型的平均绝对误差除以该目标的化学精度，小于 1 就代表达到化学精度。作者比较了五种手工设计的分子表示（各接一个现成的回归器）、两种既有的 MPNN 基准，以及自己最好的变体，也就是 edge network 的消息函数、set2set 读出，再加上明确的氢原子。\n\n| 方法 | 13 个目标的平均误差比 |\n| --- | ---: |\n| 分子指纹 ECFP4 | 53.97 |\n| Coulomb 矩阵 | 3.37 |\n| 化学键袋 BoB | 2.08 |\n| HDAD | 1.35 |\n| 分子图卷积 GC | 2.59 |\n| GG-NN | 1.36 |\n| **enn-s2s（本文）** | **0.68** |\n| enn-s2s，5 个模型的集成 | 0.52 |\n\n单一模型在 13 个目标上都是当时最好的，其中 11 个达到化学精度。作者也发现，每个目标各训练一个模型，比 13 个目标一起训练更好，有时差到 40%。\n\n### 没有空间信息时\n\nQM9 给了每个分子的 3D 构形，但真实的应用往往只有分子的连接方式。作者另外测了只给拓扑的设置，结果发现，凡是能让远方节点交换信息的改动都有帮助：\n\n| 模型 | 平均误差比 |\n| --- | ---: |\n| GG-NN | 3.47 |\n| GG-NN + 虚拟边 | 2.90 |\n| GG-NN + 主节点 | 2.62 |\n| GG-NN + set2set 读出 | 2.57 |\n\n三种改动在 13 个目标上都有改善，set2set 的版本在 5 个目标上达到化学精度。这组实验多用了一个部分电荷的节点特征，它本身是 DFT 的输出，实际应用时拿不到；前面那张主要结果表没有用这个特征。\n\n多塔的版本也带来一个意外的好处。作者原本只是为了加快训练，结果在两种训练设置下，它都在 13 个目标里的 12 个胜过基准，泛化也比较好。\n\n### 虚拟边与主节点\n\n这两个改动在论文里只占一小段，回头看却很有意思。三年多后，Alon 与 Yahav 把 GNN 难以传递远方信息的原因称为 over-squashing，他们的修补也是让最后一层所有节点都相连。再往后的 Graph Transformer，则是每一层都加一条让所有节点互相看见的路。\n\n\n> [!TIP]\n> \n> over-squashing 与后来把全局注意力放进每一层的做法，可以参考我们的笔记：\n> \n> - **[[20.06] Over-squashing：塞不下了](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS：图上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n## 结论\n\n这篇论文把当时各自命名的图神经网络，整理成「消息、更新、读出」三个函数。这个名字后来被沿用得非常广，许多论文直接用 message passing 指称这一整类模型。\n\n在 QM9 上，作者用 edge network、set2set 和明确的氢原子，把 13 个目标都推到当时最好，其中 11 个达到化学精度。\n\n作者在文中呼吁，与其不断提出新的变体，不如先在真正重要的应用上，把这个家族推到极限，找出哪些实现细节真的关键。读完整篇，最有用的往往也就是这些细节，例如氢原子要不要当节点、每个目标要不要分开训练、读出用哪一种。\n","imageSizes":{"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png":{"width":628,"height":388}}}