{"notes":[{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"遠方的考題","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/lrgb.be2cd7b008af.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"五個資料集","text":"五個資料集","depth":3},{"id":"基準模型","text":"基準模型","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"胜肽資料集","text":"胜肽資料集","depth":3},{"id":"超像素資料集","text":"超像素資料集","depth":3},{"id":"作者的觀察","text":"作者的觀察","depth":3},{"id":"基準本身也被重新檢查","text":"基準本身也被重新檢查","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"聽見圖的形狀","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/san.e84a43ca7dce.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"學出來的位置編碼","text":"學出來的位置編碼","depth":3},{"id":"全連接但分開兩套注意力","text":"全連接，但分開兩套注意力","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"gamma-該設多大","text":"\\gamma 該設多大？","depth":3},{"id":"全連接和-lpe-各自的貢獻","text":"全連接和 LPE 各自的貢獻","depth":3},{"id":"和當時最好的模型比","text":"和當時最好的模型比","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/over-squashing.0e8f072a16a4.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"問題半徑","text":"問題半徑","depth":3},{"id":"和-rnn-的瓶頸比較","text":"和 RNN 的瓶頸比較","depth":3},{"id":"只看傳播比例","text":"只看傳播比例","depth":3},{"id":"一個刻意設計的任務","text":"一個刻意設計的任務","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"全連接的最後一層","text":"全連接的最後一層","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物資料集","text":"生物資料集","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"圖上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphgps.9db287ae3c36.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"位置與結構","text":"位置與結構","depth":3},{"id":"為什麼訊息傳遞需要它們","text":"為什麼訊息傳遞需要它們","depth":3},{"id":"先傳遞再注意會先丟掉資訊","text":"先傳遞再注意，會先丟掉資訊","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一層-gps","text":"一層 GPS","depth":3},{"id":"線性的複雜度","text":"線性的複雜度","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"消融兩條路各自的份量","text":"消融：兩條路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年後的重新評估","text":"一年後的重新評估","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"2022 年，Graph Transformer 的論文一篇接一篇。它們的賣點都差不多，訊息傳遞的 GNN 每層只看一步的鄰居，遠方的資訊傳不過來；Transformer 讓每個節點看所有節點，長距離的關係一次就能建立。\n\n但當時常用的基準，很難證明這件事。ZINC、ogbg-mol* 這些分子資料集，一張圖平均只有二十幾個節點；在這些基準上，加了位置編碼的訊息傳遞 GNN 常常和 Transformer 打平，甚至更好，排行榜前幾名多半還是局部的模型。\n\nMila、南洋理工大學與 Valence Discovery 的 Vijay Prakash Dwivedi、Ladislav Rampášek、Mikhail Galkin 等人認為，問題出在考題。要看出模型能不能處理長距離的互動，得先有一份真的需要長距離資訊才答得出來的考卷。\n\n這份考卷叫 LRGB，Long Range Graph Benchmark。\n\n## 定義問題\n\n什麼樣的資料集算是「需要長距離」？作者提出三個判斷的特徵。\n\n**圖的大小**。訊息傳遞的 GNN 要處理距離 $r$ 的互動，至少得疊 $r$ 層，感受野會隨層數指數成長，造成 over-squashing。但如果圖本身只有二十幾個節點，$r$ 自然也小，瓶頸不明顯，局部的 GNN 就能學得很好。所以圖要夠大，才拉得開兩類模型的差距。\n\n**任務的性質**。圖大還不夠，任務本身也要依賴遠方的資訊。作者舉了兩個例子。ZINC 的任務和局部子結構的數量有關，研究顯示數到長度 7 的子結構就夠了，再長也沒有幫助。MalNet-Tiny 的圖可以有好幾千個節點，但全連接的模組帶來的進步很少。\n\n**全域結構的貢獻**。如果任務能從全域的位置編碼得到好處，或者目標本身就和距離有關，例如分子的 2D、3D 結構性質，那麼它就需要全域的資訊。\n\n## 解決問題\n\n### 五個資料集\n\n作者依這三個特徵，從電腦視覺和化學兩個領域，做了五個資料集。\n\n**PascalVOC-SP** 與 **COCO-SP** 來自影像的語意分割。先用 SLIC 演算法把影像切成超像素，每個超像素是一個節點，相鄰的超像素之間連邊（rag-boundary 圖），任務是預測每個節點的語意類別。\n\n[![原論文 Figure 2：由左到右四張圖。原始影像是幾隻玩具動物；第二張是 SLIC 切出的超像素，綠色邊界；第三張是在超像素上疊加的 rag-boundary 圖；最後一張只畫出圖本身，節點依類別上色。](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)\n\n*原論文 Figure 2（Dwivedi 等人），取自 [**PDF 第 5 頁**](https://arxiv.org/pdf/2206.08164v4#page=5)，CC BY 4.0。圖中為了方便展示，超像素少於 50 個；圖說未收錄。*\n\n以前的超像素資料集，例如 MNIST、CIFAR10，一張圖最多只有 75 和 150 個節點。這裡最多取到 500 個超像素，平均最短路徑長度是 10.74，直徑 27.62；MNIST 是 3.03 和 6.03，CIFAR10 是 3.97 和 8.46。\n\n**PCQM-Contact** 是分子上的連結預測。任務是預測哪兩個原子，在圖上相隔超過 5 步，在 3D 空間裡卻靠得很近（距離小於 3.5 Å）。5 步的門檻排除了圖上本來就很近的原子對，模型得學到和 3D 結構有關的性質。\n\n**Peptides-func** 與 **Peptides-struct** 用的是同一批 15,535 個胜肽分子。胜肽是胺基酸組成的短鏈，比蛋白質短，但比一般的藥物小分子大得多；節點是重原子，邊是化學鍵。Peptides-func 是多標籤分類，預測胜肽的功能，例如抗菌、抗病毒，共 10 類；Peptides-struct 是多標籤回歸，預測由 3D 結構算出的幾種性質，例如慣性、長度、球形度。輸入裡完全沒有 2D 或 3D 的座標。\n\n| 資料集 | 圖數 | 平均節點數 | 平均最短路徑 | 平均直徑 | 任務 |\n| --- | ---: | ---: | ---: | ---: | --- |\n| PascalVOC-SP | 11,355 | 479.40 | 10.74 | 27.62 | 節點分類 |\n| COCO-SP | 123,286 | 476.88 | 10.66 | 27.39 | 節點分類 |\n| PCQM-Contact | 529,434 | 30.14 | 4.63 | 9.86 | 連結預測 |\n| Peptides-func | 15,535 | 150.94 | 20.89 | 56.99 | 圖分類 |\n| Peptides-struct | 15,535 | 150.94 | 20.89 | 56.99 | 圖回歸 |\n\n胜肽的平均直徑將近 57，是 PCQM-Contact 的六倍左右，平均度數卻只有 2 左右，一條長鏈，資訊得一步一步傳過去。\n\n### 基準模型\n\n作者選了兩類模型。訊息傳遞的 GNN 有 GCN、GCNII、GINE 和 GatedGCN；全連接的 Transformer 有加了拉普拉斯位置編碼的 Transformer，以及改良版的 SAN。為了公平，所有模型的參數都控制在 50 萬左右，每個實驗用 4 個亂數種子。胜肽資料集上，訊息傳遞的 GNN 用 5 層，Transformer 用 4 層。\n\n\n> [!TIP]\n> \n> SAN 的學習式位置編碼與兩套注意力，可以參考我們的筆記：\n> \n> - **[[21.06] SAN：聽見圖的形狀](/blog/papers/graph-learning/san)**\n\n\n## 討論\n\n### 胜肽資料集\n\n| 模型 | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 0.5930 | 0.3496 |\n| GCNII | 0.5543 | 0.3471 |\n| GINE | 0.5498 | 0.3547 |\n| GatedGCN | 0.5864 | 0.3420 |\n| GatedGCN + RWSE | 0.6069 | 0.3357 |\n| Transformer + LapPE | 0.6326 | **0.2529** |\n| SAN + LapPE | 0.6384 | 0.2683 |\n| SAN + RWSE | **0.6439** | 0.2545 |\n\nTransformer 類的模型在兩個胜肽資料集上都明顯領先，Peptides-struct 上的差距最大，MAE 從 0.34 左右降到 0.25 左右。\n\n### 超像素資料集\n\nPascalVOC-SP 上，訊息傳遞的 GNN 用 8 層，F1 分數 GCN 是 0.1268、GINE 是 0.1265、GatedGCN 是 0.2873；Transformer + LapPE 是 0.2694，SAN + LapPE 是 0.3230。COCO-SP 上，Transformer 和 GatedGCN 的差距不大；SAN 在 A100 上 60 小時的時限內沒有收斂。\n\n### 作者的觀察\n\n作者從結果整理出幾點：\n\n1. **簡單的局部 GNN 表現差**。GCN 和 GINE 除了 Peptides-func 以外都表現不好，這和 over-squashing 論文的發現一致，GCN、GIN 比注意力式的 GNN 更受瓶頸影響。\n2. **太淺的 GNN 學不起來**。只用 2 層的訊息傳遞 GNN 明顯欠擬合。這和 Cora、Citeseer 這類經典基準相反，在那些資料上淺的 GCN 反而比深的好。\n3. **全連接的 Transformer 最好**，差距在胜肽資料集最明顯。作者把原因歸於任務的長距離性質、非局部資訊的貢獻，以及圖的大小。\n4. **規模是 Transformer 的弱點**。COCO-SP 上 SAN 跑不完，顯示當時 Transformer 類模型在大圖上的擴展性還有問題。\n\n同一批作者的 GraphGPS 論文，就拿這份基準當長距離的測試，在五個資料集中的四個勝出。\n\n\n> [!TIP]\n> \n> GraphGPS 的架構，以及它想處理的 over-squashing，可以參考我們的筆記：\n> \n> - **[[20.06] Over-squashing：塞不下了](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS：圖上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n### 基準本身也被重新檢查\n\n一年後，RWTH Aachen 的 Tönshoff 等人重新跑了這份基準。他們沒有改模型，只替訊息傳遞的 GNN 做了基本的超參數搜尋，加上視覺資料集的特徵正規化，Transformer 和 GNN 之間的差距就大幅縮小，在胜肽資料集上甚至消失。他們也指出 PCQM-Contact 的評估程式碼和論文定義不一致。\n\nLRGB 的作者在 2023 年 11 月更新的論文版本裡，加了一段話，請讀者參考這份重新評估。\n\n\n> [!TIP]\n> \n> 重新評估的細節，可以參考我們的筆記：\n> \n> - **[[23.09] Where Did the Gap Go?：差距去哪了？](/blog/papers/graph-learning/where-did-the-gap-go)**\n\n\n## 結論\n\nLRGB 先回答「什麼樣的資料集算長距離」，用圖的大小、任務的性質和全域結構的貢獻三個特徵，從超像素影像和胜肽分子做出五個資料集。在作者的基準實驗裡，局部的訊息傳遞 GNN 明顯落後全連接的 Transformer，胜肽資料集上差距最大。\n\n這份考卷後來成了 Graph Transformer 論文的標準考場。讀它的結果表時，最好把 Tönshoff 等人的重新評估放在旁邊一起看；同一個 GCN，在兩份表裡的 Peptides-func 分數差了將近 0.09。\n","imageSizes":{"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png":{"width":1098,"height":214}}}