{"notes":[{"id":"graph-learning/mpnn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"MPNN","subtitle":"原來都在傳話","author":"Z. Yuan","paperDate":"2017-04-04","publishedOn":"2023-10-11","world":"external/mpnn","paperTitle":"Neural Message Passing for Quantum Chemistry","paperUrl":"https://arxiv.org/abs/1704.01212","bibliography":{"title":"Neural Message Passing for Quantum Chemistry","authors":["Justin Gilmer","Samuel S. Schoenholz","Patrick F. Riley","Oriol Vinyals","George E. Dahl"],"url":"https://arxiv.org/abs/1704.01212","pdfUrl":"https://arxiv.org/pdf/1704.01212v2","codeUrl":null,"preprint":{"id":"1704.01212","submittedOn":"2017-04-04","url":"https://arxiv.org/abs/1704.01212"},"publication":{"venue":"ICML","year":2017,"kind":"conference","url":"https://proceedings.mlr.press/v70/gilmer17a.html"},"citation":"@misc{graphlearningmpnn2017,\n  title = {{Neural Message Passing for Quantum Chemistry}},\n  author = {Justin Gilmer and Samuel S. Schoenholz and Patrick F. Riley and Oriol Vinyals and George E. Dahl},\n  year = {2017},\n  eprint = {1704.01212},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.01212}\n}\n","citationPath":"/content/papers/citations/graph-learning/mpnn.5c9aab7ce227.bib"},"thumbnail":{"src":"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png","width":628,"height":388},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/mpnn.8686540a7216.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"訊息傳遞的框架","text":"訊息傳遞的框架","depth":3},{"id":"作者試了哪些變化","text":"作者試了哪些變化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"qm9-的主要結果","text":"QM9 的主要結果","depth":3},{"id":"沒有空間資訊時","text":"沒有空間資訊時","depth":3},{"id":"虛擬邊與主節點","text":"虛擬邊與主節點","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/over-squashing.0e8f072a16a4.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"問題半徑","text":"問題半徑","depth":3},{"id":"和-rnn-的瓶頸比較","text":"和 RNN 的瓶頸比較","depth":3},{"id":"只看傳播比例","text":"只看傳播比例","depth":3},{"id":"一個刻意設計的任務","text":"一個刻意設計的任務","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"全連接的最後一層","text":"全連接的最後一層","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物資料集","text":"生物資料集","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"圖上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphgps.9db287ae3c36.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"位置與結構","text":"位置與結構","depth":3},{"id":"為什麼訊息傳遞需要它們","text":"為什麼訊息傳遞需要它們","depth":3},{"id":"先傳遞再注意會先丟掉資訊","text":"先傳遞再注意，會先丟掉資訊","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一層-gps","text":"一層 GPS","depth":3},{"id":"線性的複雜度","text":"線性的複雜度","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"消融兩條路各自的份量","text":"消融：兩條路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年後的重新評估","text":"一年後的重新評估","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"要知道一個分子的偶極矩、能隙或零點振動能，化學家通常得跑一次密度泛函理論（DFT）的計算。一個小分子大約要一千秒。\n\n如果藥物或材料的篩選要看上百萬個候選分子，這個速度就成了瓶頸。\n\nGoogle Brain、Google 與 DeepMind 的 Justin Gilmer、Samuel Schoenholz、Patrick Riley、Oriol Vinyals 與 George Dahl 想用神經網路直接從分子圖預測這些性質，一個分子只要百分之一秒左右。\n\n當時已經有不少能吃分子圖的神經網路，各自取了不同的名字。這篇論文先做的事，是把它們放進同一個框架裡。\n\n[![原論文 Figure 1：一個有機分子有兩條路通往性質預測。上面的路是 DFT 計算，約需 10 的 3 次方秒；下面的路是訊息傳遞神經網路，原子之間一步步交換訊息，約需 10 的負 2 次方秒，兩條路的輸出都是能量、振動頻率等目標值。](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)\n\n*原論文 Figure 1（Gilmer 等人），取自 [**PDF 第 1 頁**](https://arxiv.org/pdf/1704.01212v2#page=1)。圖說未收錄。*\n\n## 定義問題\n\n作者指出，機器學習在化學上的應用，多數還停在特徵工程。研究者先手工設計分子的表示法，例如 Coulomb 矩陣、化學鍵袋（Bag of Bonds）、分子指紋，再接一個現成的回歸模型。\n\n神經網路已經有人試過，但還沒被廣泛採用。作者把這比作卷積網路普及以前的影像模型，缺的是證據，證明帶有適當歸納偏差的神經網路，在這個領域真的有用。\n\n分子有天然的對稱性。把原子重新編號，分子還是同一個分子；適合它的模型，對圖的同構應該不變。作者想做兩件事，一是整理出現有圖神經網路的共同形式，二是把這個形式在一個實際的基準上推到極限。\n\n基準是 QM9。約 13 萬個小分子，每個分子有 13 種用 DFT 算出的性質。評估時用一個很直觀的標準，化學界訂的「化學精度」；誤差低於它，就等於達到化學家能接受的準確度。\n\n## 解決問題\n\n### 訊息傳遞的框架\n\nMPNN（Message Passing Neural Networks）把前向傳遞分成兩個階段。\n\n**訊息傳遞階段**跑 $T$ 步。每一步，節點 $v$ 從每個鄰居 $w$ 收到一則訊息，加總之後更新自己的狀態：\n\n$$\nm_v^{t+1}=\\sum_{w\\in N(v)}M_t\\left(h_v^t,h_w^t,e_{vw}\\right)\n$$\n\n$$\nh_v^{t+1}=U_t\\left(h_v^t,m_v^{t+1}\\right)\n$$\n\n$M_t$ 是訊息函數，看得到收發雙方的狀態和邊的特徵 $e_{vw}$；$U_t$ 是更新函數。\n\n**讀出階段**把所有節點的最終狀態合成整張圖的表示：\n\n$$\n\\hat y=R\\left(\\left\\{h_v^T\\mid v\\in G\\right\\}\\right)\n$$\n\n$R$ 必須對節點的排列不變。三個函數都是可學習的。\n\n作者接著逐一說明，至少八個既有的模型都能寫成這個形式，包括 Duvenaud 等人的分子指紋卷積、門控圖神經網路（GG-NN）、Interaction Networks、分子圖卷積、深度張量網路，以及三個以拉普拉斯為基礎的頻域方法（Bruna 等人、ChebNet、GCN）。\n\n以 GCN 為例，訊息函數是「鄰居的狀態乘上權重，再依兩端的度數縮放」，更新函數是 ReLU。拿一張五個節點的小圖，用訊息與更新函數的寫法算一次，再和 GCN 的矩陣寫法對照：\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\nn = 5\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]\nnbrs = {v: [v] for v in range(n)}            # 每個節點先放自己：自環也算一條邊\nfor i, j in edges:\n    nbrs[i].append(j)\n    nbrs[j].append(i)\ndeg = {v: len(nbrs[v]) for v in range(n)}    # 加上自環後的度數\nh = rng.normal(size=(n, 3))\nW = rng.normal(size=(3, 3))\n\ndef message(h_v, h_w, v, w):                 # M_t(h_v, h_w, e_vw)\n    return h_w @ W / np.sqrt(deg[v] * deg[w])\n\ndef update(h_v, m_v):                        # U_t(h_v, m_v)\n    return np.maximum(m_v, 0)\n\ndef mpnn_step(h):\n    return np.array([update(h[v], sum(message(h[v], h[w], v, w) for w in nbrs[v]))\n                     for v in range(n)])\n\n# 對照 GCN 的矩陣寫法 ReLU(Â H W)\nA = np.eye(n)\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nprint(\"和 GCN 的矩陣寫法相同:\", np.allclose(mpnn_step(h), np.maximum(A_hat @ h @ W, 0)))\n```\n\n```text\n和 GCN 的矩陣寫法相同: True\n```\n\n這兩種寫法算的是同一件事，只是一個從單一節點的角度描述，一個把整張圖寫成矩陣。框架的好處在於，換一個 `message` 或 `update`，就是另一個模型。\n\n### 作者試了哪些變化\n\n有了共同的形式，作者在每個元件上各試幾種選擇：\n\n- **訊息函數**：GG-NN 的作法是依邊的類型挑一個矩陣乘上鄰居狀態。作者提出 **edge network**，用一個神經網路把連續的邊特徵向量映射成 $d\\times d$ 的矩陣，這樣原子間距離這類連續值也能放進來。另一種 **pair message** 讓訊息同時看收發雙方的狀態，但訓練比較不穩定，作者沒有繼續深入。\n- **虛擬的圖元素**：一種是在沒有相連的節點對之間，加上一種特殊類型的「虛擬邊」；另一種是加一個和所有節點相連的**主節點**，當作每一步大家都能讀寫的全域暫存區。兩者都讓資訊在傳遞階段走得更遠。\n- **讀出函數**：除了 GG-NN 的加總式讀出，作者也試了為集合設計的 set2set，表達力應該更強。\n- **多塔（towers）**：把 $d$ 維的節點狀態切成 $k$ 份，各自做訊息傳遞，再用一個共享的網路混合。同樣的計算量下，可以用更大的隱藏狀態。\n\n輸入方面，作者發現把氫原子當成明確的節點、放進完整的邊特徵（鍵的類型和原子間距離），對很多目標都很重要。\n\n## 討論\n\n### QM9 的主要結果\n\n評估指標是**誤差比**，模型的平均絕對誤差除以該目標的化學精度，小於 1 就代表達到化學精度。作者比較了五種手工設計的分子表示（各接一個現成的回歸器）、兩種既有的 MPNN 基準，以及自己最好的變體，也就是 edge network 的訊息函數、set2set 讀出，再加上明確的氫原子。\n\n| 方法 | 13 個目標的平均誤差比 |\n| --- | ---: |\n| 分子指紋 ECFP4 | 53.97 |\n| Coulomb 矩陣 | 3.37 |\n| 化學鍵袋 BoB | 2.08 |\n| HDAD | 1.35 |\n| 分子圖卷積 GC | 2.59 |\n| GG-NN | 1.36 |\n| **enn-s2s（本文）** | **0.68** |\n| enn-s2s，5 個模型的集成 | 0.52 |\n\n單一模型在 13 個目標上都是當時最好的，其中 11 個達到化學精度。作者也發現，每個目標各訓練一個模型，比 13 個目標一起訓練更好，有時差到 40%。\n\n### 沒有空間資訊時\n\nQM9 給了每個分子的 3D 構形，但真實的應用往往只有分子的連接方式。作者另外測了只給拓撲的設定，結果發現，凡是能讓遠方節點交換資訊的改動都有幫助：\n\n| 模型 | 平均誤差比 |\n| --- | ---: |\n| GG-NN | 3.47 |\n| GG-NN + 虛擬邊 | 2.90 |\n| GG-NN + 主節點 | 2.62 |\n| GG-NN + set2set 讀出 | 2.57 |\n\n三種改動在 13 個目標上都有改善，set2set 的版本在 5 個目標上達到化學精度。這組實驗多用了一個部分電荷的節點特徵，它本身是 DFT 的輸出，實際應用時拿不到；前面那張主要結果表沒有用這個特徵。\n\n多塔的版本也帶來一個意外的好處。作者原本只是為了加快訓練，結果在兩種訓練設定下，它都在 13 個目標裡的 12 個勝過基準，泛化也比較好。\n\n### 虛擬邊與主節點\n\n這兩個改動在論文裡只占一小段，回頭看卻很有意思。三年多後，Alon 與 Yahav 把 GNN 難以傳遞遠方資訊的原因稱為 over-squashing，他們的修補也是讓最後一層所有節點都相連。再往後的 Graph Transformer，則是每一層都加一條讓所有節點互相看見的路。\n\n\n> [!TIP]\n> \n> over-squashing 與後來把全域注意力放進每一層的做法，可以參考我們的筆記：\n> \n> - **[[20.06] Over-squashing：塞不下了](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS：圖上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n## 結論\n\n這篇論文把當時各自命名的圖神經網路，整理成「訊息、更新、讀出」三個函數。這個名字後來被沿用得非常廣，許多論文直接用 message passing 指稱這一整類模型。\n\n在 QM9 上，作者用 edge network、set2set 和明確的氫原子，把 13 個目標都推到當時最好，其中 11 個達到化學精度。\n\n作者在文中呼籲，與其不斷提出新的變體，不如先在真正重要的應用上，把這個家族推到極限，找出哪些實作細節真的關鍵。讀完整篇，最有用的往往也就是這些細節，例如氫原子要不要當節點、每個目標要不要分開訓練、讀出用哪一種。\n","imageSizes":{"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png":{"width":628,"height":388}}}