{"notes":[{"id":"graph-learning/where-did-the-gap-go","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Where Did the Gap Go?","subtitle":"Where Did the Gap Go?","author":"Z. Yuan","paperDate":"2023-09-01","publishedOn":"2025-11-12","paperTitle":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2309.00367","bibliography":{"title":"Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark","authors":["Jan Tönshoff","Martin Ritzert","Eran Rosenbluth","Martin Grohe"],"url":"https://arxiv.org/abs/2309.00367","pdfUrl":"https://arxiv.org/pdf/2309.00367v2","codeUrl":"https://github.com/toenshoff/LRGB","preprint":{"id":"2309.00367","submittedOn":"2023-09-01","url":"https://arxiv.org/abs/2309.00367"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2024,"kind":"journal","url":"https://openreview.net/forum?id=Nm0WX86sKv"},"citation":"@misc{graphlearningwheredidthegapgo2023,\n  title = {{Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark}},\n  author = {Jan Tönshoff and Martin Ritzert and Eran Rosenbluth and Martin Grohe},\n  year = {2023},\n  eprint = {2309.00367},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2309.00367}\n}\n","citationPath":"/content/papers/citations/graph-learning/where-did-the-gap-go.288771073197.bib"},"thumbnail":{"src":"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png","width":1118,"height":416},"locale":"en","path":"/content/papers/en/graph-learning/where-did-the-gap-go.d761c8afe4b4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"basic-hyperparameter-search","text":"Basic Hyperparameter Search","depth":3},{"id":"channel-wise-normalization","text":"Channel-Wise Normalization","depth":3},{"id":"filtered-mrr","text":"Filtered MRR","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"peptide-datasets","text":"Peptide Datasets","depth":3},{"id":"one-prediction-head","text":"One Prediction Head","depth":3},{"id":"superpixel-datasets","text":"Superpixel Datasets","depth":3},{"id":"pcqm-contact","text":"PCQM-Contact","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/lrgb","relation":"referenced","topic":""},{"id":"graph-learning/gnn-plus","relation":"referenced","topic":""}]},{"id":"graph-learning/lrgb","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"LRGB","subtitle":"Tests from Afar","author":"Z. Yuan","paperDate":"2022-06-16","publishedOn":"2025-10-16","world":"external/lrgb","paperTitle":"Long Range Graph Benchmark","paperUrl":"https://arxiv.org/abs/2206.08164","bibliography":{"title":"Long Range Graph Benchmark","authors":["Vijay Prakash Dwivedi","Ladislav Rampášek","Mikhail Galkin","Ali Parviz","Guy Wolf","Anh Tuan Luu","Dominique Beaini"],"url":"https://arxiv.org/abs/2206.08164","pdfUrl":"https://arxiv.org/pdf/2206.08164v4","codeUrl":"https://github.com/vijaydwivedi75/lrgb","preprint":{"id":"2206.08164","submittedOn":"2022-06-16","url":"https://arxiv.org/abs/2206.08164"},"publication":{"venue":"NeurIPS Datasets and Benchmarks","shortVenue":"NeurIPS D&B","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_and_Benchmarks.html"},"citation":"@misc{graphlearninglrgb2022,\n  title = {{Long Range Graph Benchmark}},\n  author = {Vijay Prakash Dwivedi and Ladislav Rampášek and Mikhail Galkin and Ali Parviz and Guy Wolf and Anh Tuan Luu and Dominique Beaini},\n  year = {2022},\n  eprint = {2206.08164},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2206.08164}\n}\n","citationPath":"/content/papers/citations/graph-learning/lrgb.4f1c69bdfc3e.bib"},"thumbnail":{"src":"/content/papers/assets/1c4286bdcce2817b9755835b80dd77ae2c30adc996a6665ccf8f7be2ba7c964f.png","width":1098,"height":214},"locale":"en","path":"/content/papers/en/graph-learning/lrgb.e5726d9cee8c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"five-datasets","text":"Five Datasets","depth":3},{"id":"baseline-models","text":"Baseline Models","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"peptide-datasets","text":"Peptide Datasets","depth":3},{"id":"superpixel-datasets","text":"Superpixel Datasets","depth":3},{"id":"the-authors-observations","text":"The Authors' Observations","depth":3},{"id":"reexamining-the-benchmark-itself","text":"Reexamining the Benchmark Itself","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/gnn-plus","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GNN+","subtitle":"Old Steeds, New Ambitions","author":"Z. Yuan","paperDate":"2025-02-13","publishedOn":"2025-12-06","paperTitle":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","paperUrl":"https://arxiv.org/abs/2502.09263","bibliography":{"title":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","authors":["Yuankai Luo","Lei Shi","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/2502.09263","pdfUrl":"https://arxiv.org/pdf/2502.09263v3","codeUrl":"https://github.com/LUOyk1999/GNNPlus","preprint":{"id":"2502.09263","submittedOn":"2025-02-13","url":"https://arxiv.org/abs/2502.09263"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/luo25h.html"},"citation":"@misc{graphlearninggnnplus2025,\n  title = {{Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence}},\n  author = {Yuankai Luo and Lei Shi and Xiao-Ming Wu},\n  year = {2025},\n  eprint = {2502.09263},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.09263}\n}\n","citationPath":"/content/papers/citations/graph-learning/gnn-plus.7d14d260e648.bib"},"thumbnail":{"src":"/content/papers/assets/0be1b8a606332519d042698f1c6afd3ef8857eb95682b95df1cdeb9e3bfe7648.png","width":604,"height":510},"locale":"en","path":"/content/papers/en/graph-learning/gnn-plus.e317f7610577.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"two-kinds-of-task","text":"Two Kinds of Task","depth":3},{"id":"is-the-comparison-fair","text":"Is the Comparison Fair?","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-classic-models","text":"Three Classic Models","depth":3},{"id":"six-techniques","text":"Six Techniques","depth":3},{"id":"how-to-compare","text":"How to Compare","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"overall-results","text":"Overall Results","depth":3},{"id":"faster-too","text":"Faster Too","depth":3},{"id":"what-each-technique-does","text":"What Each Technique Does","depth":3},{"id":"the-authors-own-limitations","text":"The Authors' Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/classic-gnns","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""},{"id":"graph-learning/where-did-the-gap-go","relation":"referenced","topic":""}]}],"text":"The 2022 Long Range Graph Benchmark (LRGB) contains a persuasive table. On long-range peptide datasets, message passing GCN, GINE and GatedGCN all clearly lose to Transformers. Many later Graph Transformer papers use this table as a baseline, usually reusing LRGB's code and reported GNN numbers.\n\nJan Tönshoff, Eran Rosenbluth and Martin Grohe at RWTH Aachen, together with Martin Ritzert at Göttingen, retune these baselines. They propose no new model; they simply tune the existing ones properly.\n\nOn several datasets, the gap in the title disappears.\n\n\n> [!TIP]\n> \n> For LRGB's five datasets and original experiments, see our note:\n> \n> - **[[22.06] LRGB: Tests from Afar](/blog/papers/graph-learning/lrgb)**\n\n\n## Defining the Problem\n\nThe authors raise three concerns about LRGB.\n\n**Hyperparameters.** The reported GNN settings were not carefully tuned and cannot represent these models' capabilities.\n\n**Feature normalization.** PascalVOC-SP and COCO-SP node and edge features have channels differing by orders of magnitude in scale. Feeding these directly into neural networks can produce poorly conditioned activations. Channel-wise normalization is standard in computer vision, yet almost absent in LRGB and later work using it, except CRaWl.\n\n**Link-prediction metrics.** PCQM-Contact uses mean reciprocal rank (MRR). The paper defines filtered MRR, but the supplied code computes raw MRR.\n\nThe first two make GNNs look worse than they are; the third makes results across papers incomparable without checking evaluation settings.\n\n## Solving the Problem\n\n### Basic Hyperparameter Search\n\nWithin LRGB's 500,000-parameter budget and predefined ranges, the authors search major GCN, GINE and GatedGCN hyperparameters, including depth and dropout. Positional/structural encoding—none, LapPE or RWSE—is also selected as a hyperparameter.\n\nThey reassess GraphGPS the same way, improving it substantially on three datasets too. Both sides receive tuning.\n\n### Channel-Wise Normalization\n\nFor PascalVOC-SP and COCO-SP, compute each channel's training-set mean $\\mu_i$ and standard deviation $\\sigma_i$, then feed $\\tilde x_i=(x_i-\\mu_i)/\\sigma_i$ to the model.\n\n### Filtered MRR\n\nFor a true edge $(h, t)$, rank $t$ among all candidate $(h, x)$ pairs by score, take its reciprocal rank, then average.\n\nA head may have multiple true tails. If another true tail scores higher, it pushes $t$ down despite the prediction also being correct. **Filtered** evaluation removes other true tails before ranking.\n\nThe authors notice another issue. LRGB scores with a symmetric inner product; a node's self-product is often highest, putting self-loop $(h, h)$ first. Self-loops are meaningless here and never correct, so **extended filtering** removes them too.\n\nTry a head with two true edges t1 and t2, itself, and two incorrect candidates:\n\n```python\nscores = {\"h self\": 0.95, \"t1\": 0.90, \"t2\": 0.80, \"x1\": 0.60, \"x2\": 0.40}\ntrue_tails = [\"t1\", \"t2\"]                     # This head has two true edges\n\ndef mrr(drop):\n    total = 0\n    for t in true_tails:\n        candidates = [c for c in scores if c == t or c not in drop(t)]\n        rank = 1 + sum(scores[c] > scores[t] for c in candidates)\n        total += 1 / rank\n    return total / len(true_tails)\n\nprint(\"raw            :\", round(mrr(lambda t: set()), 3))\nprint(\"filtered       :\", round(mrr(lambda t: set(true_tails)), 3))         # Remove other true tails\nprint(\"extended filter:\", round(mrr(lambda t: set(true_tails) | {\"h self\"}), 3))   # Remove self-loops too\n```\n\n```text\nraw            : 0.417\nfiltered       : 0.5\nextended filter: 1.0\n```\n\nThe same scores yield three MRRs. In raw evaluation, t2 ranks third behind t1 and self; filtering removes t1 and leaves it second; removing self makes both true tails rank first. The model stays identical while the number changes from 0.417 to 1.0.\n\n## Discussion\n\n### Peptide Datasets\n\n[![Figure 1 of the paper: the left table compares Peptides-func AP and Peptides-struct MAE in three groups—original LRGB, retuned results with green relative improvements, and newer models. On the right, bars compare GatedGCN, GINE and GCN with a linear head, a two-layer MLP head and full tuning; a red line marks the GPS baseline.](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)](/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png)\n\n*Figure 1 from Tönshoff et al., taken from [**PDF page 2**](https://arxiv.org/pdf/2309.00367v2#page=2), CC BY 4.0. Subcaptions and the original caption are omitted.*\n\nThe table has three sections: original LRGB numbers at the top, retuned results in the middle, and newer models below.\n\n| Model | Peptides-func AP ↑ | Peptides-struct MAE ↓ |\n| --- | ---: | ---: |\n| GCN (original LRGB) | 0.5930 | 0.3496 |\n| GPS (GraphGPS paper) | 0.6535 | 0.2500 |\n| GCN (tuned) | **0.6860** (+16%) | **0.2460** (+30%) |\n| GINE (tuned) | 0.6621 (+20%) | 0.2473 (+30%) |\n| GatedGCN (tuned) | 0.6765 (+11%) | 0.2477 (+26%) |\n| GPS (reassessed) | 0.6534 | 0.2509 |\n\nAll three tuned GNNs beat GPS on both peptide datasets. GCN's Peptides-func 0.6860 matches newer Graph Transformers such as GRIT and Graph ViT; Peptides-struct 0.2460 ties the best result of the time. The authors write that a simple GCN suffices on Peptides-Struct. GPS remains near its original numbers, which they consider its actual performance level.\n\n### One Prediction Head\n\nThe key hyperparameter turns out to be **prediction-head depth**.\n\nThe right-hand ablation changes LRGB's single linear head to a two-layer MLP, keeping other hyperparameters fixed. Blue is linear, orange two-layer MLP, green full tuning and red the GPS reference. Changing the head already helps Peptides-func substantially; on Peptides-struct, it nearly closes the entire GNN–Graph Transformer gap. GPS barely changes with the head.\n\nThe authors suggest the targets depend nonlinearly on global graph information. A linear head after message passing cannot represent that function, while Graph Transformers process global information through deep feedforward networks at every layer and depend less on the final linear head. A deeper prediction head is, they argue, a simpler and cheaper solution.\n\n### Superpixel Datasets\n\n| Model | PascalVOC-SP F1 ↑ | COCO-SP F1 ↑ |\n| --- | ---: | ---: |\n| GCN (original LRGB) | 0.1268 | 0.0841 |\n| GatedGCN (original LRGB) | 0.2873 | 0.2641 |\n| GPS (GraphGPS paper) | 0.3748 | 0.3412 |\n| GCN (normalization + tuning) | 0.2078 (+64%) | 0.1338 (+59%) |\n| GINE (normalization + tuning) | 0.2718 (+115%) | 0.2125 (+59%) |\n| GatedGCN (normalization + tuning) | 0.3880 (+35%) | 0.2922 (+11%) |\n| GPS (normalization + tuning) | **0.4440** (+18%) | **0.3884** (+13%) |\n\nEvery model improves substantially. On PascalVOC-SP, GatedGCN reaches 0.3880, above original GPS's 0.3748; GPS itself rises to 0.4440, within roughly one percentage point of CRaWl, then the only normalized model. The authors attribute most of the original GPS–CRaWl gap to GPS consuming unnormalized raw signals.\n\nAblations show normalization alone accounts for over half each model's improvement, except GatedGCN. GatedGCN's accuracy changes little, but variation across seeds falls sharply.\n\nThese two superpixel datasets are the only ones retaining a clear Graph Transformer lead after tuning.\n\n### PCQM-Contact\n\nRetuning improves raw MRR by roughly 3 percentage points, erasing GPS's earlier slight lead.\n\nFiltered MRR adds roughly another 3 percentage points for the same predictions. Comparing that with raw-MRR literature values can easily look like a large model improvement. Extended filtering removes self-loops and adds around 10 more percentage points, also changing the ranking: GPS regains a small lead. The authors suspect GPS predictions are particularly penalized by self-loops.\n\nThe evaluation choice directly changes model rankings.\n\n## Conclusion\n\nThis short paper does three things. Basic hyperparameter search erases the peptide gap, with a two-layer prediction head the key. Feature normalization improves all visual-dataset models. A mismatch between PCQM-Contact's evaluation code and definition is identified, and filtered and extended-filtered procedures are provided.\n\nThe conclusion leaves an open question. Only two superpixel datasets retain a clear Graph Transformer advantage. Perhaps the other tasks can be solved without long-range interactions, or Graph Transformers have no inherent advantage in exploiting long-range dependencies.\n\nTwo years later, Luo et al. at Beihang and Hong Kong Polytechnic University extend this idea to 14 graph-level datasets. Six common techniques let classic GNNs match or beat Graph Transformers.\n\n\n> [!TIP]\n> \n> For systematically strengthening classic GNNs and comparing them with Graph Transformers, see our note:\n> \n> - **[[25.02] GNN+: Old Steeds, New Ambitions](/blog/papers/graph-learning/gnn-plus)**\n\n\nA baseline table reused by many papers is rewritten by a two-layer prediction head. In a comparison table, baseline settings deserve as close a look as the new model's design.\n","imageSizes":{"/content/papers/assets/2381a94a5409a041eaab5b06f8dd33ebdd448087d689d89d780486cf3b900353.png":{"width":1118,"height":416}}}