{"notes":[{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/over-squashing.0e8f072a16a4.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"問題半徑","text":"問題半徑","depth":3},{"id":"和-rnn-的瓶頸比較","text":"和 RNN 的瓶頸比較","depth":3},{"id":"只看傳播比例","text":"只看傳播比例","depth":3},{"id":"一個刻意設計的任務","text":"一個刻意設計的任務","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"全連接的最後一層","text":"全連接的最後一層","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物資料集","text":"生物資料集","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"抹得太平了","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/deeper-insights.8ca58f8b9ca8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"先和全連接網路比","text":"先和全連接網路比","depth":3},{"id":"拉普拉斯平滑","text":"拉普拉斯平滑","depth":3},{"id":"平滑太多次","text":"平滑太多次","depth":3},{"id":"太淺也不行","text":"太淺也不行","depth":3},{"id":"補救讓-gcn-多看到一些標籤","text":"補救：讓 GCN 多看到一些標籤","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"圖上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphgps.9db287ae3c36.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"位置與結構","text":"位置與結構","depth":3},{"id":"為什麼訊息傳遞需要它們","text":"為什麼訊息傳遞需要它們","depth":3},{"id":"先傳遞再注意會先丟掉資訊","text":"先傳遞再注意，會先丟掉資訊","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一層-gps","text":"一層 GPS","depth":3},{"id":"線性的複雜度","text":"線性的複雜度","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"消融兩條路各自的份量","text":"消融：兩條路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年後的重新評估","text":"一年後的重新評估","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"圖神經網路從 2005 年前後被提出以來，就一直有個老問題，遠方節點的資訊很難傳過來。\n\n實務上，GNN 疊超過幾層就不再進步，大家接受的解釋是 over-smoothing，層數一多，所有節點的表示都變得一樣。Technion 的 Uri Alon 與 Eran Yahav 指出，over-smoothing 大多是在論文分類這類短距離任務上觀察到的，這類任務的答案主要取決於附近的鄰居，本來就不需要太多層。任務如果真的需要遠方的資訊，問題可能出在別的地方。\n\n\n> [!TIP]\n> \n> over-smoothing 的出處與它的證明，可以參考我們的筆記：\n> \n> - **[[18.01] Deeper Insights：抹得太平了](/blog/papers/graph-learning/deeper-insights)**\n\n\n他們注意到，要讓距離 $r$ 的節點互相影響，至少得疊 $r$ 層；而每多疊一層，一個節點要照顧的鄰居數量就呈指數成長，這些資訊全都得塞進同一個固定長度的向量。他們把這個現象叫做 **over-squashing**，意思是資訊被過度擠壓。\n\n## 定義問題\n\n### 問題半徑\n\n作者先定義**問題半徑** $r$，也就是一個預測任務需要的互動距離。$r$ 通常事先不知道，實務上是靠調整層數 $K$ 來估計。\n\n節點 $v$ 在 $K$ 層之後的感受野 $\\mathcal N_v^K$，是它 $K$ 步以內的所有節點。如果任務需要的是長距離的互動，就必須 $K\\ge r$，否則遠方的節點根本傳不到，這叫做 **under-reaching**。\n\n但 $|\\mathcal N_v^K|$ 會隨 $K$ 指數成長。在聚合那一步，這些指數成長的資訊全被加總進一個固定長度的向量，關鍵的訊息就送不到目的地。模型只好學會訓練資料裡的短距離訊號，到了測試時就泛化得很差。\n\n### 和 RNN 的瓶頸比較\n\n這讓人想起機器翻譯在注意力出現之前的情況。seq2seq 的編碼器得把整個輸入句子壓進最後一個隱藏狀態，句子一長，資訊就丟失了。\n\n[![原論文 Figure 1：左圖是 RNN seq2seq 的瓶頸，輸入序列的資訊沿著一條鏈依序傳到最後一個節點；右圖是 GNN 的瓶頸，左側呈樹狀展開的大量節點，資訊一層一層匯集到最右邊的目標節點，紅色曲線在目標節點前擠成一束。](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)\n\n*原論文 Figure 1（Alon 與 Yahav），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2006.05205v4#page=2)。圖說未收錄。*\n\n作者認為 GNN 的瓶頸更嚴重。RNN 的感受野隨步數**線性**成長，GNN 則是**指數**成長；同樣是一個固定長度的向量，GNN 要塞的東西多得多。\n\n### 只看傳播比例\n\n拿一棵二元樹看看。只看傳播的比例，不管權重和非線性，每個葉節點經過 $r$ 層之後，能在根節點裡留下多少：\n\n```python\nimport numpy as np\n\ndef binary_tree(r):\n    n = 2 ** (r + 1) - 1                     # 根節點是 0，節點 i 的子節點是 2i+1、2i+2\n    A = np.zeros((n, n))\n    for i in range((n - 1) // 2):\n        for c in (2 * i + 1, 2 * i + 2):\n            A[i, c] = A[c, i] = 1\n    return A\n\nfor r in range(2, 8):\n    A = binary_tree(r) + np.eye(2 ** (r + 1) - 1)\n    d = A.sum(1)\n    A_hat = A / np.sqrt(np.outer(d, d))\n    share = np.linalg.matrix_power(A_hat, r)[0, -1]     # r 步之後，最後一個葉節點在根節點裡的份量\n    print(f\"r={r}  葉節點 {2 ** r:>3} 個，每個葉節點在根節點裡的份量 {share:.5f}\")\n```\n\n```text\nr=2  葉節點   4 個，每個葉節點在根節點裡的份量 0.10206\nr=3  葉節點   8 個，每個葉節點在根節點裡的份量 0.02552\nr=4  葉節點  16 個，每個葉節點在根節點裡的份量 0.00638\nr=5  葉節點  32 個，每個葉節點在根節點裡的份量 0.00159\nr=6  葉節點  64 個，每個葉節點在根節點裡的份量 0.00040\nr=7  葉節點 128 個，每個葉節點在根節點裡的份量 0.00010\n```\n\n每深一層，葉節點的數量加倍，每個葉節點在根節點裡的份量卻縮成大約四分之一。到了 $r=7$，128 個葉節點的資訊混在一起，每個只佔萬分之一左右。這個示範只是直覺，論文的論證是從向量能存下多少資訊出發。\n\n### 一個刻意設計的任務\n\n為了把瓶頸單獨拿出來看，作者設計了 **NeighborsMatch** 任務。\n\n圖裡有幾個綠色節點，每個綠色節點有不同數量的藍色鄰居，也各有一個字母標籤。目標節點也有幾個藍色鄰居，任務是預測它的標籤，答案就是「藍色鄰居數量和它一樣」的那個綠色節點的字母。每張圖的對應關係都不同，所以模型每次都得把所有綠色節點的資訊傳到目標節點、逐一比對，沒有捷徑。\n\n這個任務看起來很刻意，但作者指出，它很像程式碼裡的一類問題。一段 Python 程式宣告了好幾個變數，各有型別和值；之後某個位置該用哪個變數，得依照當下需要的型別和值，從可見的變數裡挑。這就是後面會測的 VarMisuse 任務。\n\n**Tree-NeighborsMatch** 把中間那塊結構換成一棵深度為 $r$ 的二元樹。綠色節點都是葉子，目標節點是根，所有邊都指向根。$r$ 就是問題半徑，可以直接控制擠壓的程度。模型用 $r+1$ 層，讓資訊傳到根之後，還能再過一次非線性。\n\n## 解決問題\n\n這篇論文的主要貢獻，是指出並解釋 over-squashing；作者自己說，他們的修補方法只是最簡單的一種，用來證明這個問題普遍存在、而且一直沒被處理。\n\n### 全連接的最後一層\n\n做法是把最後一層改成**全連接層**（fully-adjacent，FA）。在這一層裡，每一對節點都視為相鄰，$\\mathcal N_v:=V$；層的種類不變，也不增加任何權重。\n\n前面 $K-1$ 層照原本的圖做訊息傳遞，學到帶有結構資訊的節點表示；最後一層讓所有節點直接互動，資訊不必再一層一層擠過瓶頸。\n\n### 需要多大的向量\n\n作者也算了一個組合下界，估計要在 Tree-NeighborsMatch 上達到 100% 訓練準確率，隱藏維度 $d$ 至少要多大。下界隨 $r$ 的階乘成長，比指數還快；對 $d=32$（他們實驗用的維度）來說，問題半徑最多只能到 $r=7$。實際上更糟，梯度下降不保證找得到那個存在的解，實驗裡就算 $d=512$，也最多擬合到 $r=7$。\n\n## 討論\n\n### Tree-NeighborsMatch\n\n[![原論文 Figure 3：橫軸是問題半徑 r 從 2 到 8，縱軸是訓練準確率。GGNN、GAT、GIN、GCN 四條曲線在 r=2、3 都是 1.0；r=4 時 GGNN 和 GAT 仍為 1.0，GIN 掉到 0.77，GCN 掉到 0.70；之後全部下降，r=8 時都在 0.2 以下。](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)\n\n*原論文 Figure 3（Alon 與 Yahav），取自 [**PDF 第 5 頁**](https://arxiv.org/pdf/2006.05205v4#page=5)。圖說未收錄。*\n\n這張圖看的是**訓練**準確率，不是測試。GCN 和 GIN 只能完美擬合到 $r=3$，$r=4$ 時 GCN 只剩 70%；從 $r=5$ 開始，四種 GNN 都無法完美擬合訓練集，即使每個訓練樣本已經看過上萬次。\n\nGAT 和 GGNN 撐到 $r=4$，作者的解釋在聚合的方式。GCN 和 GIN 先把所有鄰居的訊息加總成一個向量，之後才和目標節點自己的表示結合；GAT 用注意力，可以根據目標節點的表示挑選要看的鄰居，只把相關的資訊往上傳。GGNN 也有類似的過濾，只是做在逐元素的閘門上。\n\n既然所有 GNN 在這裡都擬合不了訓練集，在公開的長距離資料集上，它們又是怎麼擬合的？作者的假設是，模型靠的是訓練集裡的短距離訊號與一些巧合的特徵，沒有真正學到被擠掉的長距離資訊，所以測試時泛化不好。\n\n### QM9\n\nQM9 有約 13 萬個分子，每個分子平均約 18 個原子，要回歸 13 種量子化學性質。作者直接拿 Brockschmidt（2020）的實作，那份工作替多種 GNN 搜尋了 500 組設定，大多數 GNN 的最佳層數是 8 層，暗示這個任務需要長距離的資訊。\n\n作者用同樣的程式、同樣的切分和設定，只把最後一層改成 FA，每個性質訓練 5 次。結果平均把錯誤率降低了 42%。\n\n他們接著排除了幾個可能的解釋：\n\n- **是不是 under-reaching？**至少 90% 的樣本，層數已經大於等於分子的直徑，遠方節點本來就傳得到。改用 10 層的模型，也沒有比原本好。\n- **是不是向量不夠大？**把維度加倍，只改善了 5.5%；FA 用原本的維度、不加權重，改善了 42%。\n- **一定要全連接嗎？**最後一層只隨機加上一半的邊，錯誤率仍然平均降低 31.5%，加的邊越多，改善越多。\n- **那乾脆全部層都全連接？**錯誤率反而高了 1500%。圖的結構還是需要的，只是最後要有一個讓資訊直接交流的地方。\n\n### 生物資料集\n\nNCI1 有 4,110 個化合物，要預測有沒有抗肺癌活性；ENZYMES 有 600 個酵素，要分成 6 類。作者沿用 Errica 等人（2020）的公平比較設定，10 折、3 個亂數種子，共 30 次的平均。\n\nErrica 等人當時發現，在 ENZYMES 上，完全不用圖結構的基準比所有 GNN 都好。加上 FA 之後，GIN 在 ENZYMES 上比原本高 8.1 個百分點，也比不用圖的基準高 2.5 個百分點；在 NCI1 上，GIN+FA 比原本最好的 GIN 高 1.5 個百分點。平均來說，FA 讓錯誤率在 ENZYMES 降低 12%、在 NCI1 降低 4.8%。\n\n在 VarMisuse 程式資料集上，五種 GNN 加上 FA 之後，在訓練時見過的專案上全都進步，R-GCN 達到 88.4%；在沒見過的專案上，多數模型也進步了，R-GCN 從 82.9% 升到 83.8%，GGNN 和 GNN-MLP 則小幅下降。\n\n## 結論\n\n這篇論文把 GNN 處理長距離資訊的困難，指向一個以前沒被單獨討論的原因。感受野指數成長，向量長度固定，資訊在瓶頸上被擠掉；GCN 和 GIN 這種一視同仁加總鄰居的模型受影響最大。一個不加任何權重的全連接最後一層，就在大量調參過的模型上帶來明顯改善。\n\n作者明說 FA 只是起點。後來的研究沿著這個方向發展出不少做法，例如改變圖的連接方式讓瓶頸變寬，或是讓每一層都同時做局部的訊息傳遞與全域的注意力，GraphGPS 就是後者。\n\n\n> [!TIP]\n> \n> 把訊息傳遞和全域注意力放在同一層的做法，可以參考我們的筆記：\n> \n> - **[[22.05] GraphGPS：圖上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n如果你的 GNN 在長距離的任務上卡住了，加深、加寬之前，先在最後一層把所有節點連起來試試。不用加任何參數，改一下那一層的鄰接矩陣就好。\n","imageSizes":{"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png":{"width":1078,"height":356},"/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png":{"width":558,"height":316}}}