{"notes":[{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"遠くからの出題","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"ja","path":"/content/papers/ja/graph-learning/lrgb.bbb008c8e288.md","headings":[{"id":"問題設定","text":"問題設定","depth":2},{"id":"解決方法","text":"解決方法","depth":2},{"id":"五つのデータセット","text":"五つのデータセット","depth":3},{"id":"ベースライン","text":"ベースライン","depth":3},{"id":"議論","text":"議論","depth":2},{"id":"ペプチド","text":"ペプチド","depth":3},{"id":"スーパーピクセル","text":"スーパーピクセル","depth":3},{"id":"著者の観察","text":"著者の観察","depth":3},{"id":"ベンチマーク自体を再検査","text":"ベンチマーク自体を再検査","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"グラフの形を聴く","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"ja","path":"/content/papers/ja/graph-learning/san.e02e8dcf7c2d.md","headings":[{"id":"問題設定","text":"問題設定","depth":2},{"id":"解決方法","text":"解決方法","depth":2},{"id":"学習する位置符号化","text":"学習する位置符号化","depth":3},{"id":"全結合に二系統の注意","text":"全結合に二系統の注意","depth":3},{"id":"議論","text":"議論","depth":2},{"id":"gamma-はどれだけ","text":"\\gamma はどれだけ？","depth":3},{"id":"全結合と-lpe-の寄与","text":"全結合と LPE の寄与","depth":3},{"id":"当時の最良と比べる","text":"当時の最良と比べる","depth":3},{"id":"代償","text":"代償","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"もう入りきらない","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"ja","path":"/content/papers/ja/graph-learning/over-squashing.40644f2f5f00.md","headings":[{"id":"問題設定","text":"問題設定","depth":2},{"id":"問題の半径","text":"問題の半径","depth":3},{"id":"rnn-のボトルネックとの比較","text":"RNN のボトルネックとの比較","depth":3},{"id":"伝播比率だけを見る","text":"伝播比率だけを見る","depth":3},{"id":"意図的に作ったタスク","text":"意図的に作ったタスク","depth":3},{"id":"解決方法","text":"解決方法","depth":2},{"id":"最後の一層を全結合に","text":"最後の一層を全結合に","depth":3},{"id":"どれだけ大きなベクトルが要るか","text":"どれだけ大きなベクトルが要るか","depth":3},{"id":"議論","text":"議論","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物データセット","text":"生物データセット","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"グラフ上の GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"ja","path":"/content/papers/ja/graph-learning/graphgps.0d5dc280bee1.md","headings":[{"id":"問題設定","text":"問題設定","depth":2},{"id":"位置と構造","text":"位置と構造","depth":3},{"id":"なぜ伝播にも必要か","text":"なぜ伝播にも必要か","depth":3},{"id":"伝播のあとで注意しても遅い","text":"伝播のあとで注意しても遅い","depth":3},{"id":"解決方法","text":"解決方法","depth":2},{"id":"三つの材料","text":"三つの材料","depth":3},{"id":"gps-の一層","text":"GPS の一層","depth":3},{"id":"線形の計算量","text":"線形の計算量","depth":3},{"id":"議論","text":"議論","depth":2},{"id":"アブレーション二経路の寄与","text":"アブレーション：二経路の寄与","depth":3},{"id":"他モデルとの比較","text":"他モデルとの比較","depth":3},{"id":"一年後の再評価","text":"一年後の再評価","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"2022 年、Graph Transformer 論文が続き、売り文句も似ていました。一層で一歩しか進まない伝播型 GNN と違い、全ノードを一度に見て遠距離の関係を作れます。\n\nしかし既存ベンチマークでは証明しにくいものでした。ZINC、ogbg-mol* は平均二十数ノードだけで、位置符号化した伝播型が同等以上となり、上位も局所モデルが多くありました。\n\nMila、南洋理工大学、Valence Discovery の Vijay Prakash Dwivedi、Ladislav Rampášek、Mikhail Galkin らは、出題に問題があると考えました。遠距離を扱えるか見るなら、それが必要な問題を用意しなければなりません。\n\nその試験が LRGB、Long Range Graph Benchmark です。\n\n## 問題設定\n\n長距離を必要とするデータには、三つの特徴があるとします。\n\n**大きさ**。距離 $r$ には最低 $r$ 層が必要で、受容野が指数的に増え over-squashing を起こします。しかし二十数ノードなら $r$ も小さく、ボトルネックが目立たず局所 GNN で十分です。両者の差を見るには大きさが必要です。\n\n**タスクの性質**。大きいだけでは足りません。ZINC は局所部分構造の個数に関わり、長さ 7 まで数えれば十分で、それ以上は無益と報告されています。数千ノードの MalNet-Tiny でも、全結合の改善はわずかです。\n\n**大域構造の寄与**。大域位置符号化で改善したり、分子の 2D、3D の構造特性のように目的が距離に関わったりするなら、大域情報が必要です。\n\n## 解決方法\n\n### 五つのデータセット\n\n三特徴から、ビジョンと化学の二分野で五つを作ります。\n\n**PascalVOC-SP**、**COCO-SP** は意味分割の画像です。SLIC でスーパーピクセルに分け、各領域をノード、隣接領域をエッジにする rag-boundary グラフで、各ノードの意味クラスを予測します。\n\n[![原論文 Figure 2：左から、玩具の動物の原画像、緑境界の SLIC スーパーピクセル、rag-boundary グラフの重ね合わせ、クラスで色分けしたグラフだけの四図。](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)\n\n*原論文 Figure 2（Dwivedi ら）、[**PDF 5 ページ**](https://arxiv.org/pdf/2206.08164v4#page=5)より、CC BY 4.0。表示用に 50 未満のスーパーピクセルにしています。原キャプションは省略。*\n\n既存の MNIST、CIFAR10 は最大 75、150 ノードです。ここでは最大 500 領域にし、平均最短経路 10.74、直径 27.62。MNIST は 3.03、6.03、CIFAR10 は 3.97、8.46 でした。\n\n**PCQM-Contact** は分子のリンク予測で、グラフでは 5 ステップ超なのに 3D で 3.5 Å 未満に近い原子ペアを予測します。5 ステップの閾値で近所を除き、3D 構造に関わる性質を学ばせます。\n\n**Peptides-func**、**Peptides-struct** は同じ 15,535 ペプチドです。アミノ酸の短鎖でタンパク質より短く、薬の小分子よりずっと大きく、重原子をノード、結合をエッジにします。func は抗菌・抗ウイルスなど 10 機能の多ラベル分類、struct は慣性、長さ、球形度など 3D 由来の多出力回帰です。2D、3D 座標は入力しません。\n\n| データセット | グラフ数 | 平均ノード数 | 平均最短経路 | 平均直径 | タスク |\n| --- | ---: | ---: | ---: | ---: | --- |\n| PascalVOC-SP | 11,355 | 479.40 | 10.74 | 27.62 | ノード分類 |\n| COCO-SP | 123,286 | 476.88 | 10.66 | 27.39 | ノード分類 |\n| PCQM-Contact | 529,434 | 30.14 | 4.63 | 9.86 | リンク予測 |\n| Peptides-func | 15,535 | 150.94 | 20.89 | 56.99 | グラフ分類 |\n| Peptides-struct | 15,535 | 150.94 | 20.89 | 56.99 | グラフ回帰 |\n\nペプチドの直径は約 57、PCQM-Contact の約六倍なのに、平均次数は約 2 です。長い鎖を一歩ずつ伝わらなければなりません。\n\n### ベースライン\n\n伝播型の GCN、GCNII、GINE、GatedGCN と、全結合の LapPE 付き Transformer、改良 SAN を選びます。約 50 万パラメーター、4 シードにそろえ、ペプチドは伝播型 5 層、Transformer 4 層です。\n\n\n> [!TIP]\n> \n> SAN の LPE と二系統の注意は、こちらのノートをどうぞ。\n> \n> - **[[21.06] SAN：グラフの形を聴く](/blog/papers/graph-learning/san)**\n\n\n## 議論\n\n### ペプチド\n\n| モデル | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 0.5930 | 0.3496 |\n| GCNII | 0.5543 | 0.3471 |\n| GINE | 0.5498 | 0.3547 |\n| GatedGCN | 0.5864 | 0.3420 |\n| GatedGCN + RWSE | 0.6069 | 0.3357 |\n| Transformer + LapPE | 0.6326 | **0.2529** |\n| SAN + LapPE | 0.6384 | 0.2683 |\n| SAN + RWSE | **0.6439** | 0.2545 |\n\nTransformer 系が両方で明確に先行し、特に struct は約 0.34 から約 0.25 へ下がります。\n\n### スーパーピクセル\n\nPascalVOC-SP の伝播型は 8 層で、F1 は GCN 0.1268、GINE 0.1265、GatedGCN 0.2873。Transformer + LapPE は 0.2694、SAN + LapPE は 0.3230 です。COCO-SP は Transformer と GatedGCN の差が小さく、SAN は A100 の 60 時間以内に収束しません。\n\n### 著者の観察\n\n結果から四点を挙げます。\n\n1. **単純な局所 GNN は弱い**。GCN、GINE は func 以外で劣り、注意型よりボトルネックの影響が強いという over-squashing の発見に一致します。\n2. **浅すぎると学べない**。二層の伝播型は明確な過少適合です。浅いほうがよい Cora、Citeseer と逆です。\n3. **全結合 Transformer が最良**。特にペプチドで顕著で、長距離性、非局所情報、大きさによるとします。\n4. **規模が Transformer の弱点**。COCO-SP の SAN は完走できず、大きなグラフへの拡張に問題があります。\n\n同じ著者陣の GraphGPS は、ここで五つ中四つに勝ちました。\n\n\n> [!TIP]\n> \n> GraphGPS と over-squashing は、こちらのノートをどうぞ。\n> \n> - **[[20.06] Over-squashing：もう入りきらない](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS：グラフ上の GPS](/blog/papers/graph-learning/graphgps)**\n\n\n### ベンチマーク自体を再検査\n\n一年後、RWTH Aachen の Tönshoff らは、モデルを変えず基本的な探索と画像特徴量の正規化を行い、差を大幅に縮めました。ペプチドでは消え、PCQM-Contact の評価コードと定義の不一致も指摘しました。\n\n2023 年 11 月の更新版は、この再評価を参照するよう一段落を加えています。\n\n\n> [!TIP]\n> \n> 詳細は、こちらのノートをどうぞ。\n> \n> - **[[23.09] Where Did the Gap Go?：差はどこへ消えた？](/blog/papers/graph-learning/where-did-the-gap-go)**\n\n\n## 結論\n\nLRGB は長距離の条件を大きさ、性質、大域構造の寄与に整理し、画像スーパーピクセルとペプチドから五セットを作りました。元の実験では局所伝播が全結合に劣り、ペプチドで最大の差がありました。\n\nGraph Transformer の標準の試験場になりましたが、表は Tönshoff らの再評価と並べて読むのがよいでしょう。同じ GCN の Peptides-func が、約 0.09 違うのです。\n","imageSizes":{"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png":{"width":1098,"height":214}}}