{"notes":[{"id":"graph-learning/where-did-the-gap-go","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Where Did the Gap Go?","subtitle":"差距去哪了？","author":"Z. Yuan","paperDate":"2023-09-01","publishedOn":"2025-11-12","paperTitle":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2309.00367","bibliography":{"title":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","authors":["Jan Tönshoff","Martin Ritzert","Eran Rosenbluth","Martin Grohe"],"url":"https://arxiv.org/abs/2309.00367","pdfUrl":"https://arxiv.org/pdf/2309.00367v2","codeUrl":"https://github.com/toenshoff/LRGB","preprint":{"id":"2309.00367","submittedOn":"2023-09-01","url":"https://arxiv.org/abs/2309.00367"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2024,"kind":"journal","url":"https://openreview.net/forum?id=Nm0WX86sKv"},"citation":"@misc{graphlearningwheredidthegapgo2023,\n  title = {{Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark}},\n  author = {Jan Tönshoff and Martin Ritzert and Eran Rosenbluth and Martin Grohe},\n  year = {2023},\n  eprint = {2309.00367},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2309.00367}\n}\n","citationPath":"/content/papers/citations/graph-learning/where-did-the-gap-go.288771073197.bib"},"thumbnail":{"src":"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png","width":1118,"height":416},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/where-did-the-gap-go.ea95f280ee9a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"基本的超參數搜尋","text":"基本的超參數搜尋","depth":3},{"id":"逐通道正規化","text":"逐通道正規化","depth":3},{"id":"filtered-mrr","text":"filtered MRR","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"胜肽資料集","text":"胜肽資料集","depth":3},{"id":"一個預測頭","text":"一個預測頭","depth":3},{"id":"超像素資料集","text":"超像素資料集","depth":3},{"id":"pcqm-contact","text":"PCQM-Contact","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/lrgb","relation":"referenced","topic":""},{"id":"graph-learning/gnn-plus","relation":"referenced","topic":""}]},{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"遠方的考題","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/lrgb.be2cd7b008af.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"五個資料集","text":"五個資料集","depth":3},{"id":"基準模型","text":"基準模型","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"胜肽資料集","text":"胜肽資料集","depth":3},{"id":"超像素資料集","text":"超像素資料集","depth":3},{"id":"作者的觀察","text":"作者的觀察","depth":3},{"id":"基準本身也被重新檢查","text":"基準本身也被重新檢查","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/gnn-plus","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GNN+","subtitle":"老驥伏櫪","author":"Z. Yuan","paperDate":"2025-02-13","publishedOn":"2025-12-06","paperTitle":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","paperUrl":"https://arxiv.org/abs/2502.09263","bibliography":{"title":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","authors":["Yuankai Luo","Lei Shi","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/2502.09263","pdfUrl":"https://arxiv.org/pdf/2502.09263v3","codeUrl":"https://github.com/LUOyk1999/GNNPlus","preprint":{"id":"2502.09263","submittedOn":"2025-02-13","url":"https://arxiv.org/abs/2502.09263"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/luo25h.html"},"citation":"@misc{graphlearninggnnplus2025,\n  title = {{Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence}},\n  author = {Yuankai Luo and Lei Shi and Xiao-Ming Wu},\n  year = {2025},\n  eprint = {2502.09263},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.09263}\n}\n","citationPath":"/content/papers/citations/graph-learning/gnn-plus.7d14d260e648.bib"},"thumbnail":{"src":"/content/papers/assets/0be1b8a606332519d042698f1c6afd3ef8857eb95682b95df1cdeb9e3bfe7648.png","width":604,"height":510},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/gnn-plus.3e66f7f0d156.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"兩種任務","text":"兩種任務","depth":3},{"id":"比較公不公平","text":"比較公不公平","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三個經典模型","text":"三個經典模型","depth":3},{"id":"六個技巧","text":"六個技巧","depth":3},{"id":"怎麼比","text":"怎麼比","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"整體結果","text":"整體結果","depth":3},{"id":"也比較快","text":"也比較快","depth":3},{"id":"每個技巧各管什麼","text":"每個技巧各管什麼","depth":3},{"id":"作者自己說的限制","text":"作者自己說的限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/classic-gnns","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""},{"id":"graph-learning/where-did-the-gap-go","relation":"referenced","topic":""}]}],"text":"2022 年的 Long Range Graph Benchmark（LRGB）有一張很有說服力的表。在需要長距離資訊的胜肽資料集上，GCN、GINE、GatedGCN 這些訊息傳遞的 GNN，全都明顯輸給 Transformer。這張表後來被許多 Graph Transformer 論文拿來當基準，而且多半直接沿用 LRGB 的程式碼與它回報的 GNN 數字。\n\nRWTH Aachen 的 Jan Tönshoff、Eran Rosenbluth、Martin Grohe 與哥廷根大學的 Martin Ritzert，把這些 GNN 基準重新調了一次。他們沒有提出新模型，只替這些基準好好調了參數。\n\n結果，標題裡的那個差距，在好幾個資料集上不見了。\n\n\n> [!TIP]\n> \n> LRGB 的五個資料集與原始的基準實驗，可以參考我們的筆記：\n> \n> - **[[22.06] LRGB：遠方的考題](/blog/papers/graph-learning/lrgb)**\n\n\n## 定義問題\n\n作者對 LRGB 提出三個疑慮。\n\n**超參數**。LRGB 回報的 GNN 結果，用的是沒有仔細調過的設定，無法代表這類模型的實力。\n\n**特徵正規化**。PascalVOC-SP 和 COCO-SP 的節點與邊特徵有好幾個通道，數值的尺度差了好幾個數量級。這樣的輸入直接丟進神經網路，很容易造成活化值的條件很差。逐通道正規化在電腦視覺裡是標準做法，但 LRGB 和之後用它的研究幾乎都沒有做，只有 CRaWl 例外。\n\n**連結預測的指標**。PCQM-Contact 用 MRR（mean reciprocal rank）評估。LRGB 論文定義的是 filtered MRR，但它提供的程式碼算的其實是 raw MRR。\n\n前兩點會讓 GNN 看起來比實際差；第三點則讓不同論文的數字無法直接比較。\n\n## 解決問題\n\n### 基本的超參數搜尋\n\n作者沿用 LRGB 規定的 50 萬參數預算，在預先定義的範圍內，替 GCN、GINE、GatedGCN 搜尋層數、dropout 等主要超參數；位置或結構編碼（不用、LapPE 或 RWSE）也當成一個超參數來選。\n\n作為參照，他們用同樣的方式重新評估 GraphGPS，它在三個資料集上也有明顯進步。所以這不是只替一方加分。\n\n### 逐通道正規化\n\n在 PascalVOC-SP 和 COCO-SP 上，作者在訓練集上算出每個特徵通道的平均 $\\mu_i$ 與標準差 $\\sigma_i$，把每個特徵改成 $\\tilde x_i=(x_i-\\mu_i)/\\sigma_i$ 再送進模型。\n\n### filtered MRR\n\n連結預測的評估方式是這樣的。給定一條真正的邊 $(h, t)$，把 $t$ 和所有候選的 $(h, x)$ 一起依分數排序，看 $t$ 排第幾名，取倒數再平均。\n\n麻煩在一個 head 可能有好幾條真正的邊。如果其他真正的 tail 分數更高，$t$ 就會被它們擠到後面，這不是模型的錯。**filtered** 的做法是排名前先把其他真正的 tail 從候選裡拿掉。\n\n作者還注意到另一件事。LRGB 用對稱的內積來打分數，節點和自己的內積通常最大，自環 $(h, h)$ 很容易排第一；但自環在這個任務裡沒有意義，也永遠不是正確答案。所以他們另外提出 **extended filtering**，連自環也從候選裡拿掉。\n\n拿一個 head 試算一次。它有兩條真正的邊 t1、t2，另外還有它自己和兩個錯誤的候選：\n\n```python\nscores = {\"h 自己\": 0.95, \"t1\": 0.90, \"t2\": 0.80, \"x1\": 0.60, \"x2\": 0.40}\ntrue_tails = [\"t1\", \"t2\"]                     # 這個 head 有兩條真正的邊\n\ndef mrr(drop):\n    total = 0\n    for t in true_tails:\n        candidates = [c for c in scores if c == t or c not in drop(t)]\n        rank = 1 + sum(scores[c] > scores[t] for c in candidates)\n        total += 1 / rank\n    return total / len(true_tails)\n\nprint(\"raw            :\", round(mrr(lambda t: set()), 3))\nprint(\"filtered       :\", round(mrr(lambda t: set(true_tails)), 3))         # 拿掉其他真正的 tail\nprint(\"extended filter:\", round(mrr(lambda t: set(true_tails) | {\"h 自己\"}), 3))   # 連自環也拿掉\n```\n\n```text\nraw            : 0.417\nfiltered       : 0.5\nextended filter: 1.0\n```\n\n同一組分數，三種算法得到三個不同的 MRR。raw 時，t2 被 t1 和自環擠到第三名；filtered 拿掉 t1 之後，t2 排第二；再拿掉自環，兩條真正的邊都排第一。模型完全沒變，數字從 0.417 變成 1.0。\n\n## 討論\n\n### 胜肽資料集\n\n[![原論文 Figure 1：左邊是 Peptides-func 的平均精確率與 Peptides-struct 的 MAE 對照表，分成 LRGB 原始數字、作者重新調參的結果（綠底標出相對進步）與其他較新的模型三組；右邊是兩張長條圖，比較 GatedGCN、GINE、GCN 用一層線性預測頭、兩層 MLP 預測頭與完整調參的結果，紅線是 GPS 的基準。](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)\n\n*原論文 Figure 1（Tönshoff 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2309.00367v2#page=2)，CC BY 4.0。子圖說與圖說未收錄。*\n\n左邊的表分三段。最上面是 LRGB 原本回報的數字，中間是作者調參後的結果，最下面是當時其他較新的模型。\n\n| 模型 | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN（LRGB 原始） | 0.5930 | 0.3496 |\n| GPS（GraphGPS 論文） | 0.6535 | 0.2500 |\n| GCN（調參後） | **0.6860**（+16%） | **0.2460**（+30%） |\n| GINE（調參後） | 0.6621（+20%） | 0.2473（+30%） |\n| GatedGCN（調參後） | 0.6765（+11%） | 0.2477（+26%） |\n| GPS（重新評估） | 0.6534 | 0.2509 |\n\n調參之後，三個 GNN 在兩個胜肽資料集上都超過 GPS。Peptides-func 上，GCN 的 0.6860 已經和 GRIT、Graph ViT 這些較新的 Graph Transformer 相當；Peptides-struct 上，GCN 的 0.2460 追平了當時最好的結果。作者寫道，在 Peptides-Struct 上，一個簡單的 GCN 就夠了。GPS 的數字則和原論文幾乎一樣，作者認為那就是它的實際水準。\n\n### 一個預測頭\n\n哪個超參數最關鍵？作者發現是**預測頭的深度**。\n\n右邊的長條圖是一組消融。把 LRGB 設定的一層線性預測頭換成兩層 MLP，其他超參數都不動。藍色是線性頭，橘色是兩層 MLP，綠色是完整調參，紅線是 GPS。Peptides-func 上，換頭已經帶來明顯的進步；Peptides-struct 上，光是換頭就幾乎補完了 GNN 和 Graph Transformer 之間的全部差距。GPS 換不換頭，表現都差不多。\n\n作者的解釋是，這兩個資料集的目標似乎和全域的圖資訊有非線性的關係。訊息傳遞的 GNN 最後只接一層線性頭，就沒辦法表示這種函數；Graph Transformer 每一層都有很深的前饋網路處理全域資訊，所以不太受線性頭影響。作者認為，換一個深一點的預測頭，是更簡單、計算也更便宜的解法。\n\n### 超像素資料集\n\n| 模型 | PascalVOC-SP F1 ↑ | COCO-SP F1 ↑ |\n| --- | ---: | ---: |\n| GCN（LRGB 原始） | 0.1268 | 0.0841 |\n| GatedGCN（LRGB 原始） | 0.2873 | 0.2641 |\n| GPS（GraphGPS 論文） | 0.3748 | 0.3412 |\n| GCN（正規化＋調參） | 0.2078（+64%） | 0.1338（+59%） |\n| GINE（正規化＋調參） | 0.2718（+115%） | 0.2125（+59%） |\n| GatedGCN（正規化＋調參） | 0.3880（+35%） | 0.2922（+11%） |\n| GPS（正規化＋調參） | **0.4440**（+18%） | **0.3884**（+13%） |\n\n所有模型都有明顯進步。PascalVOC-SP 上，GatedGCN 升到 0.3880，已經超過 GPS 原論文的 0.3748；GPS 自己也升到 0.4440，和當時唯一做了正規化的 CRaWl 只差一個百分點左右。作者說，GPS 和 CRaWl 之間原本那麼大的差距，主要是因為 GPS 吃的是沒有正規化的原始訊號。\n\n消融顯示，除了 GatedGCN 以外，光是特徵正規化就占了每個模型進步幅度的一半以上；GatedGCN 的準確率只微幅提升，但不同亂數種子之間的變異大幅縮小。\n\n這兩個超像素資料集，是調參後唯一還能看到 Graph Transformer 明顯領先的地方。\n\n### PCQM-Contact\n\n作者重新調參後，用 raw MRR 評估，絕對值大約進步 3%，原本 GPS 略微領先的現象也不見了。\n\n改用 filtered MRR，同樣一組預測，MRR 又多了約 3%；如果拿這個數字去和文獻裡用 raw MRR 算的結果比，很容易誤以為是模型的大幅進步。再用 extended filtering 拿掉自環，MRR 又多了約 10 個百分點，而且排名換了，GPS 又重新小幅領先。作者推測，GPS 的預測特別容易被自環拖累。\n\n評估方式選哪一種，會直接改變模型的排名。\n\n## 結論\n\n這篇短短的論文做了三件事。替 LRGB 的 GNN 基準做基本的超參數搜尋，胜肽資料集上的差距消失了，關鍵是一個兩層的預測頭；替視覺資料集加上特徵正規化，所有模型都進步；指出 PCQM-Contact 的評估程式和定義不一致，並補上 filtered 和 extended filtering 兩種算法。\n\n作者在結論裡留下一個開放的問題。只有兩個超像素資料集上，Graph Transformer 還有明顯的優勢；這表示其他任務也許不需要長距離的互動就能解，或者 Graph Transformer 並沒有天生比較會利用長距離的依賴。\n\n兩年後，北航與香港理工大學的 Luo 等人把同樣的想法推到 14 個圖層級的資料集，經典 GNN 補上六個常見技巧之後，和 Graph Transformer 打平甚至勝出。\n\n\n> [!TIP]\n> \n> 把經典 GNN 系統地加強、再和 Graph Transformer 比較的做法，可以參考我們的筆記：\n> \n> - **[[25.02] GNN+：老驥伏櫪](/blog/papers/graph-learning/gnn-plus)**\n\n\n一張被許多論文沿用的基準表，被一個兩層的預測頭改寫了。讀別人的比較表時，基準是怎麼設定的，和新模型是怎麼設計的，一樣值得細看。\n","imageSizes":{"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png":{"width":1118,"height":416}}}