{"notes":[{"id":"graph-learning/where-did-the-gap-go","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Where Did the Gap Go?","subtitle":"差距去哪了？","author":"Z. Yuan","paperDate":"2023-09-01","publishedOn":"2025-11-12","paperTitle":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2309.00367","bibliography":{"title":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","authors":["Jan Tönshoff","Martin Ritzert","Eran Rosenbluth","Martin Grohe"],"url":"https://arxiv.org/abs/2309.00367","pdfUrl":"https://arxiv.org/pdf/2309.00367v2","codeUrl":"https://github.com/toenshoff/LRGB","preprint":{"id":"2309.00367","submittedOn":"2023-09-01","url":"https://arxiv.org/abs/2309.00367"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2024,"kind":"journal","url":"https://openreview.net/forum?id=Nm0WX86sKv"},"citation":"@misc{graphlearningwheredidthegapgo2023,\n  title = {{Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark}},\n  author = {Jan Tönshoff and Martin Ritzert and Eran Rosenbluth and Martin Grohe},\n  year = {2023},\n  eprint = {2309.00367},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2309.00367}\n}\n","citationPath":"/content/papers/citations/graph-learning/where-did-the-gap-go.288771073197.bib"},"thumbnail":{"src":"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png","width":1118,"height":416},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/where-did-the-gap-go.ff47e75698ca.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"基本的超参数搜索","text":"基本的超参数搜索","depth":3},{"id":"逐通道归一化","text":"逐通道归一化","depth":3},{"id":"filtered-mrr","text":"filtered MRR","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"肽数据集","text":"肽数据集","depth":3},{"id":"一个预测头","text":"一个预测头","depth":3},{"id":"超像素数据集","text":"超像素数据集","depth":3},{"id":"pcqm-contact","text":"PCQM-Contact","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/lrgb","relation":"referenced","topic":""},{"id":"graph-learning/gnn-plus","relation":"referenced","topic":""}]},{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"远方的考题","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/lrgb.d8ff769b0c23.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"五个数据集","text":"五个数据集","depth":3},{"id":"基准模型","text":"基准模型","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"肽数据集","text":"肽数据集","depth":3},{"id":"超像素数据集","text":"超像素数据集","depth":3},{"id":"作者的观察","text":"作者的观察","depth":3},{"id":"基准本身也被重新检查","text":"基准本身也被重新检查","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/gnn-plus","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GNN+","subtitle":"老骥伏枥","author":"Z. Yuan","paperDate":"2025-02-13","publishedOn":"2025-12-06","paperTitle":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","paperUrl":"https://arxiv.org/abs/2502.09263","bibliography":{"title":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","authors":["Yuankai Luo","Lei Shi","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/2502.09263","pdfUrl":"https://arxiv.org/pdf/2502.09263v3","codeUrl":"https://github.com/LUOyk1999/GNNPlus","preprint":{"id":"2502.09263","submittedOn":"2025-02-13","url":"https://arxiv.org/abs/2502.09263"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/luo25h.html"},"citation":"@misc{graphlearninggnnplus2025,\n  title = {{Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence}},\n  author = {Yuankai Luo and Lei Shi and Xiao-Ming Wu},\n  year = {2025},\n  eprint = {2502.09263},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.09263}\n}\n","citationPath":"/content/papers/citations/graph-learning/gnn-plus.7d14d260e648.bib"},"thumbnail":{"src":"/content/papers/assets/0be1b8a606332519d042698f1c6afd3ef8857eb95682b95df1cdeb9e3bfe7648.png","width":604,"height":510},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/gnn-plus.15890d5dd1bb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"两种任务","text":"两种任务","depth":3},{"id":"比较公不公平","text":"比较公不公平","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三个经典模型","text":"三个经典模型","depth":3},{"id":"六个技巧","text":"六个技巧","depth":3},{"id":"怎么比","text":"怎么比","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"整体结果","text":"整体结果","depth":3},{"id":"也比较快","text":"也比较快","depth":3},{"id":"每个技巧各管什么","text":"每个技巧各管什么","depth":3},{"id":"作者自己说的限制","text":"作者自己说的限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/classic-gnns","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""},{"id":"graph-learning/where-did-the-gap-go","relation":"referenced","topic":""}]}],"text":"2022 年的 Long Range Graph Benchmark（LRGB）有一张很有说服力的表。在需要长距离信息的肽数据集上，GCN、GINE、GatedGCN 这些消息传递的 GNN，全都明显输给 Transformer。这张表后来被许多 Graph Transformer 论文拿来当基准，而且多半直接沿用 LRGB 的代码与它回报的 GNN 数字。\n\nRWTH Aachen 的 Jan Tönshoff、Eran Rosenbluth、Martin Grohe 与哥廷根大学的 Martin Ritzert，把这些 GNN 基准重新调了一次。他们没有提出新模型，只替这些基准好好调了参数。\n\n结果，标题里的那个差距，在好几个数据集上不见了。\n\n\n> [!TIP]\n> \n> LRGB 的五个数据集与原始的基准实验，可以参考我们的笔记：\n> \n> - **[[22.06] LRGB：远方的考题](/blog/papers/graph-learning/lrgb)**\n\n\n## 定义问题\n\n作者对 LRGB 提出三个疑虑。\n\n**超参数**。LRGB 回报的 GNN 结果，用的是没有仔细调过的设置，无法代表这类模型的实力。\n\n**特征归一化**。PascalVOC-SP 和 COCO-SP 的节点与边特征有好几个通道，数值的尺度差了好几个数量级。这样的输入直接丢进神经网络，很容易造成活化值的条件很差。逐通道归一化在计算机视觉里是标准做法，但 LRGB 和之后用它的研究几乎都没有做，只有 CRaWl 例外。\n\n**链接预测的指标**。PCQM-Contact 用 MRR（mean reciprocal rank）评估。LRGB 论文定义的是 filtered MRR，但它提供的代码算的其实是 raw MRR。\n\n前两点会让 GNN 看起来比实际差；第三点则让不同论文的数字无法直接比较。\n\n## 解决问题\n\n### 基本的超参数搜索\n\n作者沿用 LRGB 规定的 50 万参数预算，在预先定义的范围内，替 GCN、GINE、GatedGCN 搜索层数、dropout 等主要超参数；位置或结构编码（不用、LapPE 或 RWSE）也当成一个超参数来选。\n\n作为参照，他们用同样的方式重新评估 GraphGPS，它在三个数据集上也有明显进步。所以这不是只替一方加分。\n\n### 逐通道归一化\n\n在 PascalVOC-SP 和 COCO-SP 上，作者在训练集上算出每个特征通道的平均 $\\mu_i$ 与标准差 $\\sigma_i$，把每个特征改成 $\\tilde x_i=(x_i-\\mu_i)/\\sigma_i$ 再送进模型。\n\n### filtered MRR\n\n链接预测的评估方式是这样的。给定一条真正的边 $(h, t)$，把 $t$ 和所有候选的 $(h, x)$ 一起依分数排序，看 $t$ 排第几名，取倒数再平均。\n\n麻烦在一个 head 可能有好几条真正的边。如果其他真正的 tail 分数更高，$t$ 就会被它们挤到后面，这不是模型的错。**filtered** 的做法是排名前先把其他真正的 tail 从候选里拿掉。\n\n作者还注意到另一件事。LRGB 用对称的内积来打分数，节点和自己的内积通常最大，自环 $(h, h)$ 很容易排第一；但自环在这个任务里没有意义，也永远不是正确答案。所以他们另外提出 **extended filtering**，连自环也从候选里拿掉。\n\n拿一个 head 试算一次。它有两条真正的边 t1、t2，另外还有它自己和两个错误的候选：\n\n```python\nscores = {\"h 自己\": 0.95, \"t1\": 0.90, \"t2\": 0.80, \"x1\": 0.60, \"x2\": 0.40}\ntrue_tails = [\"t1\", \"t2\"]                     # 这个 head 有两条真正的边\n\ndef mrr(drop):\n    total = 0\n    for t in true_tails:\n        candidates = [c for c in scores if c == t or c not in drop(t)]\n        rank = 1 + sum(scores[c] > scores[t] for c in candidates)\n        total += 1 / rank\n    return total / len(true_tails)\n\nprint(\"raw            :\", round(mrr(lambda t: set()), 3))\nprint(\"filtered       :\", round(mrr(lambda t: set(true_tails)), 3))         # 拿掉其他真正的 tail\nprint(\"extended filter:\", round(mrr(lambda t: set(true_tails) | {\"h 自己\"}), 3))   # 连自环也拿掉\n```\n\n```text\nraw            : 0.417\nfiltered       : 0.5\nextended filter: 1.0\n```\n\n同一组分数，三种算法得到三个不同的 MRR。raw 时，t2 被 t1 和自环挤到第三名；filtered 拿掉 t1 之后，t2 排第二；再拿掉自环，两条真正的边都排第一。模型完全没变，数字从 0.417 变成 1.0。\n\n## 讨论\n\n### 肽数据集\n\n[![原论文 Figure 1：左边是 Peptides-func 的平均精确率与 Peptides-struct 的 MAE 对照表，分成 LRGB 原始数字、作者重新调参的结果（绿底标出相对进步）与其他较新的模型三组；右边是两张长条图，比较 GatedGCN、GINE、GCN 用一层线性预测头、两层 MLP 预测头与完整调参的结果，红线是 GPS 的基准。](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)\n\n*原论文 Figure 1（Tönshoff 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2309.00367v2#page=2)，CC BY 4.0。子图注与图注未收录。*\n\n左边的表分三段。最上面是 LRGB 原本回报的数字，中间是作者调参后的结果，最下面是当时其他较新的模型。\n\n| 模型 | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN（LRGB 原始） | 0.5930 | 0.3496 |\n| GPS（GraphGPS 论文） | 0.6535 | 0.2500 |\n| GCN（调参后） | **0.6860**（+16%） | **0.2460**（+30%） |\n| GINE（调参后） | 0.6621（+20%） | 0.2473（+30%） |\n| GatedGCN（调参后） | 0.6765（+11%） | 0.2477（+26%） |\n| GPS（重新评估） | 0.6534 | 0.2509 |\n\n调参之后，三个 GNN 在两个肽数据集上都超过 GPS。Peptides-func 上，GCN 的 0.6860 已经和 GRIT、Graph ViT 这些较新的 Graph Transformer 相当；Peptides-struct 上，GCN 的 0.2460 追平了当时最好的结果。作者写道，在 Peptides-Struct 上，一个简单的 GCN 就够了。GPS 的数字则和原论文几乎一样，作者认为那就是它的实际水准。\n\n### 一个预测头\n\n哪个超参数最关键？作者发现是**预测头的深度**。\n\n右边的长条图是一组消融。把 LRGB 设置的一层线性预测头换成两层 MLP，其他超参数都不动。蓝色是线性头，橘色是两层 MLP，绿色是完整调参，红线是 GPS。Peptides-func 上，换头已经带来明显的进步；Peptides-struct 上，光是换头就几乎补完了 GNN 和 Graph Transformer 之间的全部差距。GPS 换不换头，表现都差不多。\n\n作者的解释是，这两个数据集的目标似乎和全局的图信息有非线性的关系。消息传递的 GNN 最后只接一层线性头，就没办法表示这种函数；Graph Transformer 每一层都有很深的前馈网络处理全局信息，所以不太受线性头影响。作者认为，换一个深一点的预测头，是更简单、计算也更便宜的解法。\n\n### 超像素数据集\n\n| 模型 | PascalVOC-SP F1 ↑ | COCO-SP F1 ↑ |\n| --- | ---: | ---: |\n| GCN（LRGB 原始） | 0.1268 | 0.0841 |\n| GatedGCN（LRGB 原始） | 0.2873 | 0.2641 |\n| GPS（GraphGPS 论文） | 0.3748 | 0.3412 |\n| GCN（归一化＋调参） | 0.2078（+64%） | 0.1338（+59%） |\n| GINE（归一化＋调参） | 0.2718（+115%） | 0.2125（+59%） |\n| GatedGCN（归一化＋调参） | 0.3880（+35%） | 0.2922（+11%） |\n| GPS（归一化＋调参） | **0.4440**（+18%） | **0.3884**（+13%） |\n\n所有模型都有明显进步。PascalVOC-SP 上，GatedGCN 升到 0.3880，已经超过 GPS 原论文的 0.3748；GPS 自己也升到 0.4440，和当时唯一做了归一化的 CRaWl 只差一个百分点左右。作者说，GPS 和 CRaWl 之间原本那么大的差距，主要是因为 GPS 吃的是没有归一化的原始信号。\n\n消融显示，除了 GatedGCN 以外，光是特征归一化就占了每个模型进步幅度的一半以上；GatedGCN 的准确率只微幅提升，但不同随机数种子之间的变异大幅缩小。\n\n这两个超像素数据集，是调参后唯一还能看到 Graph Transformer 明显领先的地方。\n\n### PCQM-Contact\n\n作者重新调参后，用 raw MRR 评估，绝对值大约进步 3%，原本 GPS 略微领先的现象也不见了。\n\n改用 filtered MRR，同样一组预测，MRR 又多了约 3%；如果拿这个数字去和文献里用 raw MRR 算的结果比，很容易误以为是模型的大幅进步。再用 extended filtering 拿掉自环，MRR 又多了约 10 个百分点，而且排名换了，GPS 又重新小幅领先。作者推测，GPS 的预测特别容易被自环拖累。\n\n评估方式选哪一种，会直接改变模型的排名。\n\n## 结论\n\n这篇短短的论文做了三件事。替 LRGB 的 GNN 基准做基本的超参数搜索，肽数据集上的差距消失了，关键是一个两层的预测头；替视觉数据集加上特征归一化，所有模型都进步；指出 PCQM-Contact 的评估程序和定义不一致，并补上 filtered 和 extended filtering 两种算法。\n\n作者在结论里留下一个开放的问题。只有两个超像素数据集上，Graph Transformer 还有明显的优势；这表示其他任务也许不需要长距离的交互就能解，或者 Graph Transformer 并没有天生比较会利用长距离的依赖。\n\n两年后，北航与香港理工大学的 Luo 等人把同样的想法推到 14 个图层级的数据集，经典 GNN 补上六个常见技巧之后，和 Graph Transformer 打平甚至胜出。\n\n\n> [!TIP]\n> \n> 把经典 GNN 系统地加强、再和 Graph Transformer 比较的做法，可以参考我们的笔记：\n> \n> - **[[25.02] GNN+：老骥伏枥](/blog/papers/graph-learning/gnn-plus)**\n\n\n一张被许多论文沿用的基准表，被一个两层的预测头改写了。读别人的比较表时，基准是怎么设置的，和新模型是怎么设计的，一样值得细看。\n","imageSizes":{"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png":{"width":1118,"height":416}}}