{"notes":[{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"远方的考题","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/lrgb.d8ff769b0c23.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"五个数据集","text":"五个数据集","depth":3},{"id":"基准模型","text":"基准模型","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"肽数据集","text":"肽数据集","depth":3},{"id":"超像素数据集","text":"超像素数据集","depth":3},{"id":"作者的观察","text":"作者的观察","depth":3},{"id":"基准本身也被重新检查","text":"基准本身也被重新检查","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"听见图的形状","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/san.2705d642b9c9.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"学出来的位置编码","text":"学出来的位置编码","depth":3},{"id":"全连接但分开两套注意力","text":"全连接，但分开两套注意力","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"gamma-该设多大","text":"\\gamma 该设多大？","depth":3},{"id":"全连接和-lpe-各自的贡献","text":"全连接和 LPE 各自的贡献","depth":3},{"id":"和当时最好的模型比","text":"和当时最好的模型比","depth":3},{"id":"代价","text":"代价","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/over-squashing.1bfa8bae9091.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"问题半径","text":"问题半径","depth":3},{"id":"和-rnn-的瓶颈比较","text":"和 RNN 的瓶颈比较","depth":3},{"id":"只看传播比例","text":"只看传播比例","depth":3},{"id":"一个刻意设计的任务","text":"一个刻意设计的任务","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"全连接的最后一层","text":"全连接的最后一层","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物数据集","text":"生物数据集","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"图上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphgps.74aedb083e17.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"位置与结构","text":"位置与结构","depth":3},{"id":"为什么消息传递需要它们","text":"为什么消息传递需要它们","depth":3},{"id":"先传递再注意会先丢掉信息","text":"先传递再注意，会先丢掉信息","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一层-gps","text":"一层 GPS","depth":3},{"id":"线性的复杂度","text":"线性的复杂度","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"消融两条路各自的份量","text":"消融：两条路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年后的重新评估","text":"一年后的重新评估","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"2022 年，Graph Transformer 的论文一篇接一篇。它们的卖点都差不多，消息传递的 GNN 每层只看一步的邻居，远方的信息传不过来；Transformer 让每个节点看所有节点，长距离的关系一次就能创建。\n\n但当时常用的基准，很难证明这件事。ZINC、ogbg-mol* 这些分子数据集，一张图平均只有二十几个节点；在这些基准上，加了位置编码的消息传递 GNN 常常和 Transformer 打平，甚至更好，排行榜前几名多半还是局部的模型。\n\nMila、南洋理工大学与 Valence Discovery 的 Vijay Prakash Dwivedi、Ladislav Rampášek、Mikhail Galkin 等人认为，问题出在考题。要看出模型能不能处理长距离的交互，得先有一份真的需要长距离信息才答得出来的考卷。\n\n这份考卷叫 LRGB，Long Range Graph Benchmark。\n\n## 定义问题\n\n什么样的数据集算是「需要长距离」？作者提出三个判断的特征。\n\n**图的大小**。消息传递的 GNN 要处理距离 $r$ 的交互，至少得叠 $r$ 层，感受野会随层数指数成长，造成 over-squashing。但如果图本身只有二十几个节点，$r$ 自然也小，瓶颈不明显，局部的 GNN 就能学得很好。所以图要够大，才拉得开两类模型的差距。\n\n**任务的性质**。图大还不够，任务本身也要依赖远方的信息。作者举了两个例子。ZINC 的任务和局部子结构的数量有关，研究显示数到长度 7 的子结构就够了，再长也没有帮助。MalNet-Tiny 的图可以有好几千个节点，但全连接的模块带来的进步很少。\n\n**全局结构的贡献**。如果任务能从全局的位置编码得到好处，或者目标本身就和距离有关，例如分子的 2D、3D 结构性质，那么它就需要全局的信息。\n\n## 解决问题\n\n### 五个数据集\n\n作者依这三个特征，从计算机视觉和化学两个领域，做了五个数据集。\n\n**PascalVOC-SP** 与 **COCO-SP** 来自图像的语义分割。先用 SLIC 算法把图像切成超像素，每个超像素是一个节点，相邻的超像素之间连边（rag-boundary 图），任务是预测每个节点的语义类别。\n\n[![原论文 Figure 2：由左到右四张图。原始图像是几只玩具动物；第二张是 SLIC 切出的超像素，绿色边界；第三张是在超像素上叠加的 rag-boundary 图；最后一张只画出图本身，节点依类别上色。](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)](/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png)\n\n*原论文 Figure 2（Dwivedi 等人），取自 [**PDF 第 5 页**](https://arxiv.org/pdf/2206.08164v4#page=5)，CC BY 4.0。图中为了方便展示，超像素少于 50 个；图注未收录。*\n\n以前的超像素数据集，例如 MNIST、CIFAR10，一张图最多只有 75 和 150 个节点。这里最多取到 500 个超像素，平均最短路径长度是 10.74，直径 27.62；MNIST 是 3.03 和 6.03，CIFAR10 是 3.97 和 8.46。\n\n**PCQM-Contact** 是分子上的链接预测。任务是预测哪两个原子，在图上相隔超过 5 步，在 3D 空间里却靠得很近（距离小于 3.5 Å）。5 步的门槛排除了图上本来就很近的原子对，模型得学到和 3D 结构有关的性质。\n\n**Peptides-func** 与 **Peptides-struct** 用的是同一批 15,535 个肽分子。肽是氨基酸组成的短链，比蛋白质短，但比一般的药物小分子大得多；节点是重原子，边是化学键。Peptides-func 是多标签分类，预测肽的功能，例如抗菌、抗病毒，共 10 类；Peptides-struct 是多标签回归，预测由 3D 结构算出的几种性质，例如惯性、长度、球形度。输入里完全没有 2D 或 3D 的坐标。\n\n| 数据集 | 图数 | 平均节点数 | 平均最短路径 | 平均直径 | 任务 |\n| --- | ---: | ---: | ---: | ---: | --- |\n| PascalVOC-SP | 11,355 | 479.40 | 10.74 | 27.62 | 节点分类 |\n| COCO-SP | 123,286 | 476.88 | 10.66 | 27.39 | 节点分类 |\n| PCQM-Contact | 529,434 | 30.14 | 4.63 | 9.86 | 链接预测 |\n| Peptides-func | 15,535 | 150.94 | 20.89 | 56.99 | 图分类 |\n| Peptides-struct | 15,535 | 150.94 | 20.89 | 56.99 | 图回归 |\n\n肽的平均直径将近 57，是 PCQM-Contact 的六倍左右，平均度数却只有 2 左右，一条长链，信息得一步一步传过去。\n\n### 基准模型\n\n作者选了两类模型。消息传递的 GNN 有 GCN、GCNII、GINE 和 GatedGCN；全连接的 Transformer 有加了拉普拉斯位置编码的 Transformer，以及改良版的 SAN。为了公平，所有模型的参数都控制在 50 万左右，每个实验用 4 个随机数种子。肽数据集上，消息传递的 GNN 用 5 层，Transformer 用 4 层。\n\n\n> [!TIP]\n> \n> SAN 的学习式位置编码与两套注意力，可以参考我们的笔记：\n> \n> - **[[21.06] SAN：听见图的形状](/blog/papers/graph-learning/san)**\n\n\n## 讨论\n\n### 肽数据集\n\n| 模型 | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 0.5930 | 0.3496 |\n| GCNII | 0.5543 | 0.3471 |\n| GINE | 0.5498 | 0.3547 |\n| GatedGCN | 0.5864 | 0.3420 |\n| GatedGCN + RWSE | 0.6069 | 0.3357 |\n| Transformer + LapPE | 0.6326 | **0.2529** |\n| SAN + LapPE | 0.6384 | 0.2683 |\n| SAN + RWSE | **0.6439** | 0.2545 |\n\nTransformer 类的模型在两个肽数据集上都明显领先，Peptides-struct 上的差距最大，MAE 从 0.34 左右降到 0.25 左右。\n\n### 超像素数据集\n\nPascalVOC-SP 上，消息传递的 GNN 用 8 层，F1 分数 GCN 是 0.1268、GINE 是 0.1265、GatedGCN 是 0.2873；Transformer + LapPE 是 0.2694，SAN + LapPE 是 0.3230。COCO-SP 上，Transformer 和 GatedGCN 的差距不大；SAN 在 A100 上 60 小时的时限内没有收敛。\n\n### 作者的观察\n\n作者从结果整理出几点：\n\n1. **简单的局部 GNN 表现差**。GCN 和 GINE 除了 Peptides-func 以外都表现不好，这和 over-squashing 论文的发现一致，GCN、GIN 比注意力式的 GNN 更受瓶颈影响。\n2. **太浅的 GNN 学不起来**。只用 2 层的消息传递 GNN 明显欠拟合。这和 Cora、Citeseer 这类经典基准相反，在那些数据上浅的 GCN 反而比深的好。\n3. **全连接的 Transformer 最好**，差距在肽数据集最明显。作者把原因归于任务的长距离性质、非局部信息的贡献，以及图的大小。\n4. **规模是 Transformer 的弱点**。COCO-SP 上 SAN 跑不完，显示当时 Transformer 类模型在大图上的扩展性还有问题。\n\n同一批作者的 GraphGPS 论文，就拿这份基准当长距离的测试，在五个数据集中的四个胜出。\n\n\n> [!TIP]\n> \n> GraphGPS 的架构，以及它想处理的 over-squashing，可以参考我们的笔记：\n> \n> - **[[20.06] Over-squashing：塞不下了](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS：图上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n### 基准本身也被重新检查\n\n一年后，RWTH Aachen 的 Tönshoff 等人重新跑了这份基准。他们没有改模型，只替消息传递的 GNN 做了基本的超参数搜索，加上视觉数据集的特征归一化，Transformer 和 GNN 之间的差距就大幅缩小，在肽数据集上甚至消失。他们也指出 PCQM-Contact 的评估代码和论文定义不一致。\n\nLRGB 的作者在 2023 年 11 月更新的论文版本里，加了一段话，请读者参考这份重新评估。\n\n\n> [!TIP]\n> \n> 重新评估的细节，可以参考我们的笔记：\n> \n> - **[[23.09] Where Did the Gap Go?：差距去哪了？](/blog/papers/graph-learning/where-did-the-gap-go)**\n\n\n## 结论\n\nLRGB 先回答「什么样的数据集算长距离」，用图的大小、任务的性质和全局结构的贡献三个特征，从超像素图像和肽分子做出五个数据集。在作者的基准实验里，局部的消息传递 GNN 明显落后全连接的 Transformer，肽数据集上差距最大。\n\n这份考卷后来成了 Graph Transformer 论文的标准考场。读它的结果表时，最好把 Tönshoff 等人的重新评估放在旁边一起看；同一个 GCN，在两份表里的 Peptides-func 分数差了将近 0.09。\n","imageSizes":{"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png":{"width":1098,"height":214}}}