{"notes":[{"id":"graph-learning/gin","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GIN","subtitle":"Summing Is What Counts","author":"Z. Yuan","paperDate":"2018-10-01","publishedOn":"2023-12-28","world":"external/gin","paperTitle":"How Powerful are Graph Neural Networks?","paperUrl":"https://arxiv.org/abs/1810.00826","bibliography":{"title":"How Powerful are Graph Neural Networks?","authors":["Keyulu Xu","Weihua Hu","Jure Leskovec","Stefanie Jegelka"],"url":"https://arxiv.org/abs/1810.00826","pdfUrl":"https://arxiv.org/pdf/1810.00826v3","codeUrl":"https://github.com/weihua916/powerful-gnns","preprint":{"id":"1810.00826","submittedOn":"2018-10-01","url":"https://arxiv.org/abs/1810.00826"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.00826"},"citation":"@misc{graphlearninggin2018,\n  title = {{How Powerful are Graph Neural Networks?}},\n  author = {Keyulu Xu and Weihua Hu and Jure Leskovec and Stefanie Jegelka},\n  year = {2018},\n  eprint = {1810.00826},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.00826}\n}\n","citationPath":"/content/papers/citations/graph-learning/gin.f2b31912b576.bib"},"thumbnail":{"src":"/content/papers/assets/1062d104bcaae50e5f8c9abf6c72f574a014c68e6a7bb4c6705f663336773b4f.png","width":960,"height":214},"locale":"en","path":"/content/papers/en/graph-learning/gin.8710e40a579a.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"a-common-form-for-neighbor-aggregation","text":"A Common Form for Neighbor Aggregation","depth":3},{"id":"neighbors-form-a-multiset","text":"Neighbors Form a Multiset","depth":3},{"id":"the-wl-test","text":"The WL Test","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-ceiling-for-gnns","text":"The Ceiling for GNNs","depth":3},{"id":"sum-mean-and-max","text":"Sum, Mean and Max","depth":3},{"id":"gin","text":"GIN","depth":3},{"id":"weaker-can-still-be-useful","text":"Weaker Can Still Be Useful","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"can-it-fit-the-training-set","text":"Can It Fit the Training Set?","depth":3},{"id":"test-accuracy","text":"Test Accuracy","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/sgc","relation":"citing","topic":""},{"id":"graph-learning/gnn-plus","relation":"citing","topic":""}]},{"id":"graph-learning/sgc","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SGC","subtitle":"Is Linear Enough?","author":"Z. Yuan","paperDate":"2019-02-19","publishedOn":"2024-02-09","world":"external/sgc","paperTitle":"Simplifying Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1902.07153","bibliography":{"title":"Simplifying Graph Convolutional Networks","authors":["Felix Wu","Tianyi Zhang","Amauri Holanda de Souza","Christopher Fifty","Tao Yu","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1902.07153","pdfUrl":"https://arxiv.org/pdf/1902.07153v2","codeUrl":"https://github.com/Tiiiger/SGC","preprint":{"id":"1902.07153","submittedOn":"2019-02-19","url":"https://arxiv.org/abs/1902.07153"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/wu19e.html"},"citation":"@misc{graphlearningsgc2019,\n  title = {{Simplifying Graph Convolutional Networks}},\n  author = {Felix Wu and Tianyi Zhang and Amauri Holanda de Souza and Christopher Fifty and Tao Yu and Kilian Q. Weinberger},\n  year = {2019},\n  eprint = {1902.07153},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.07153}\n}\n","citationPath":"/content/papers/citations/graph-learning/sgc.5b3815dbf17c.bib"},"thumbnail":{"src":"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png","width":1366,"height":612},"locale":"en","path":"/content/papers/en/graph-learning/sgc.b9d6518d0273.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"remove-the-nonlinearities","text":"Remove the Nonlinearities","depth":3},{"id":"a-spectral-view-a-low-pass-filter","text":"A Spectral View: A Low-Pass Filter","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"citation-networks-and-reddit","text":"Citation Networks and Reddit","depth":3},{"id":"how-much-faster","text":"How Much Faster?","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gin","relation":"referenced","topic":""}]},{"id":"graph-learning/gnn-plus","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GNN+","subtitle":"Old Steeds, New Ambitions","author":"Z. Yuan","paperDate":"2025-02-13","publishedOn":"2025-12-06","paperTitle":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","paperUrl":"https://arxiv.org/abs/2502.09263","bibliography":{"title":"Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence","authors":["Yuankai Luo","Lei Shi","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/2502.09263","pdfUrl":"https://arxiv.org/pdf/2502.09263v3","codeUrl":"https://github.com/LUOyk1999/GNNPlus","preprint":{"id":"2502.09263","submittedOn":"2025-02-13","url":"https://arxiv.org/abs/2502.09263"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/luo25h.html"},"citation":"@misc{graphlearninggnnplus2025,\n  title = {{Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence}},\n  author = {Yuankai Luo and Lei Shi and Xiao-Ming Wu},\n  year = {2025},\n  eprint = {2502.09263},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.09263}\n}\n","citationPath":"/content/papers/citations/graph-learning/gnn-plus.7d14d260e648.bib"},"thumbnail":{"src":"/content/papers/assets/0be1b8a606332519d042698f1c6afd3ef8857eb95682b95df1cdeb9e3bfe7648.png","width":604,"height":510},"locale":"en","path":"/content/papers/en/graph-learning/gnn-plus.e317f7610577.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"two-kinds-of-task","text":"Two Kinds of Task","depth":3},{"id":"is-the-comparison-fair","text":"Is the Comparison Fair?","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-classic-models","text":"Three Classic Models","depth":3},{"id":"six-techniques","text":"Six Techniques","depth":3},{"id":"how-to-compare","text":"How to Compare","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"overall-results","text":"Overall Results","depth":3},{"id":"faster-too","text":"Faster Too","depth":3},{"id":"what-each-technique-does","text":"What Each Technique Does","depth":3},{"id":"the-authors-own-limitations","text":"The Authors' Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/classic-gnns","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""},{"id":"graph-learning/where-did-the-gap-go","relation":"referenced","topic":""}]}],"text":"By 2018, graph-neural-network variants are too numerous to count. GCN, GraphSAGE and GAT each aggregate differently. New papers mostly establish superiority through experiments; few explain why, or which graph structures they cannot distinguish.\n\nKeyulu Xu and Stefanie Jegelka at MIT, together with Weihua Hu and Jure Leskovec at Stanford, change the question. Before asking about accuracy, ask whether a GNN can map two different graphs to different representations.\n\nThe title asks how powerful graph neural networks are. The answer has a ceiling, set by an algorithm from 1968.\n\n## Defining the Problem\n\n### A Common Form for Neighbor Aggregation\n\nThe authors first write existing GNNs in a common form. At layer $k$, each node $v$ aggregates its neighbors' representations and combines them with its own:\n\n$$\na_v^{(k)}=\\mathrm{AGGREGATE}^{(k)}\\left(\\left\\{h_u^{(k-1)}:u\\in\\mathcal N(v)\\right\\}\\right),\\qquad\nh_v^{(k)}=\\mathrm{COMBINE}^{(k)}\\left(h_v^{(k-1)},a_v^{(k)}\\right)\n$$\n\nFor graph classification, a final READOUT combines all node representations into one graph representation. GraphSAGE aggregates with max pooling and concatenation; GCN averages self and neighbors, then applies weights and ReLU.\n\n### Neighbors Form a Multiset\n\nThe key observation is that neighboring features form a **multiset**, where a value may appear multiple times. Two blue neighbors and three blue neighbors are different multisets.\n\nIf an aggregator maps two different multisets to the same result, the GNN cannot distinguish those neighborhoods. So the question of GNN power reduces to whether its aggregation function is **injective** over multisets.\n\n### The WL Test\n\nGraph isomorphism offers the most direct measure: isomorphic graphs should share a representation, and non-isomorphic graphs should differ. No polynomial-time algorithm is known for this problem, so the authors use a weaker but practical standard: the **Weisfeiler–Lehman (WL) test**.\n\nWL repeatedly recolors nodes. Initially all nodes have one color; each round maps a node's current color together with the multiset of neighboring colors to a new color. If two graphs have different color counts in any round, they are non-isomorphic.\n\nIn code, recolor both graphs together so their color IDs remain comparable:\n\n```python\nfrom collections import Counter\n\ndef wl(graphs, rounds=3):\n    \"\"\"Run WL coloring jointly on graphs, sharing color IDs for comparison.\"\"\"\n    nbrs = []\n    for n, edges in graphs:\n        adj = [[] for _ in range(n)]\n        for i, j in edges:\n            adj[i].append(j)\n            adj[j].append(i)\n        nbrs.append(adj)\n    colors = [[0] * n for n, _ in graphs]           # All nodes start with one color\n    for r in range(rounds):\n        sigs = [[(c[v], tuple(sorted(c[u] for u in adj[v]))) for v in range(len(c))]\n                for c, adj in zip(colors, nbrs)]    # Self color plus multiset of neighboring colors\n        table = {s: k for k, s in enumerate(sorted({s for g in sigs for s in g}))}\n        colors = [[table[s] for s in g] for g in sigs]\n        print(f\"Round {r + 1}:\", [sorted(Counter(c).items()) for c in colors])\n\npath = (4, [(0, 1), (1, 2), (2, 3)])          # A path\nstar = (4, [(0, 1), (0, 2), (0, 3)])          # A star\nwl([path, star])\n```\n\n```text\nRound 1: [[(0, 2), (1, 2)], [(0, 3), (2, 1)]]\nRound 2: [[(0, 2), (2, 2)], [(1, 3), (3, 1)]]\nRound 3: [[(0, 2), (2, 2)], [(1, 3), (3, 1)]]\n```\n\nEach round prints each graph's pairs of (color, count). They differ in the first round: the path has two endpoints and two internal nodes, while the star has three leaves and a center. WL identifies them as non-isomorphic.\n\n## Solving the Problem\n\n### The Ceiling for GNNs\n\nLemma 2 states that whenever a neighbor-aggregation GNN maps two graphs to different representations, WL can also distinguish them. **Such GNNs are at most as discriminative as WL.**\n\nConversely, Theorem 3 states that if aggregation, combination and READOUT are all injective, a sufficiently deep GNN can match WL.\n\nThe reasoning is intuitive. Each WL round is an injective hash over multisets; an injective GNN aggregator can imitate it round by round. It can also map similar substructures to nearby vectors, something discrete colors cannot do.\n\n### Sum, Mean and Max\n\nWhich aggregators are injective? The authors prove that **sum** can represent injective multiset functions, whereas common **mean** and **max** aggregators cannot.\n\n[![Figure 2 of the paper: the input multiset contains two red balls and four blue balls, followed by the information preserved by sum, mean and max. Sum keeps the full multiset, mean keeps red/blue proportions, and max keeps only the presence of red and blue.](/content/papers/assets/1062d104bcaae50e5f8c9abf6c72f574a014c68e6a7bb4c6705f663336773b4f.png)](/content/papers/assets/1062d104bcaae50e5f8c9abf6c72f574a014c68e6a7bb4c6705f663336773b4f.png)\n\n*Figure 2 from Xu et al., taken from [**PDF page 6**](https://arxiv.org/pdf/1810.00826v3#page=6). The original caption is omitted.*\n\n- **Sum** captures the full multiset.\n- **Mean** captures element proportions. $\\{a, b\\}$ and $\\{a, a, b, b\\}$ have the same mean.\n- **Max** captures only which elements appear, regardless of repetition.\n\nRepresent two neighbor types with two-dimensional one-hot vectors and compare:\n\n```python\nimport numpy as np\na, b = np.array([1.0, 0.0]), np.array([0.0, 1.0])\ncases = {\"{a, a} vs {a, a, a}\": ([a, a], [a, a, a]),\n         \"{a, b} vs {a, a, b, b}\": ([a, b], [a, a, b, b]),\n         \"{a, b} vs {a, b, b}\": ([a, b], [a, b, b])}\nfor name, (X1, X2) in cases.items():\n    same = {agg: np.allclose(f(np.array(X1), axis=0), f(np.array(X2), axis=0))\n            for agg, f in [(\"sum\", np.sum), (\"mean\", np.mean), (\"max\", np.max)]}\n    print(f\"{name:<24}\", {k: \"indistinguishable\" if v else \"distinguishable\" for k, v in same.items()})\n```\n\n```text\n{a, a} vs {a, a, a}      {'sum': 'distinguishable', 'mean': 'indistinguishable', 'max': 'indistinguishable'}\n{a, b} vs {a, a, b, b}   {'sum': 'distinguishable', 'mean': 'indistinguishable', 'max': 'indistinguishable'}\n{a, b} vs {a, b, b}      {'sum': 'distinguishable', 'mean': 'distinguishable', 'max': 'indistinguishable'}\n```\n\nThese examples correspond to the three cases in Figure 3:\n\n[![Figure 3 of the paper: (a) v has two same-color neighbors and v' has three; neither mean nor max distinguishes them. (b) v has one green and one red neighbor, v' one green and two red; max fails. (c) v has one green and one red, v' two green and two red; both mean and max fail.](/content/papers/assets/33c4b052156c190bc85033ddb47e50c70b500425938ac89108383fc039b88837.png)](/content/papers/assets/33c4b052156c190bc85033ddb47e50c70b500425938ac89108383fc039b88837.png)\n\n*Figure 3 from Xu et al., taken from [**PDF page 6**](https://arxiv.org/pdf/1810.00826v3#page=6). The main caption is omitted.*\n\nCase (a) deserves a closer look. When every node has identical features, mean and max always return that value, regardless of neighbor count. The graph structure becomes entirely invisible.\n\n### GIN\n\nUsing sum, the authors combine aggregation and combination into one step, with a multilayer perceptron (MLP) learning the injective function:\n\n$$\nh_v^{(k)}=\\mathrm{MLP}^{(k)}\\left(\\left(1+\\epsilon^{(k)}\\right)\\cdot h_v^{(k-1)}+\\sum_{u\\in\\mathcal N(v)}h_u^{(k-1)}\\right)\n$$\n\n$\\epsilon$ may be learned or fixed. Multiplying self by $1+\\epsilon$ keeps self distinct from neighbors. Corollary 6 proves this is injective over the pair (self, neighbor multiset) for many choices of $\\epsilon$, including irrational values. The experimental version with $\\epsilon=0$ is called GIN-0.\n\nFor graph classification, node representations are summed separately at each layer, then concatenated across all layers:\n\n$$\nh_G=\\mathrm{CONCAT}\\left(\\mathrm{READOUT}\\left(\\left\\{h_v^{(k)}\\mid v\\in G\\right\\}\\right)\\ \\middle|\\ k=0,1,\\dots,K\\right)\n$$\n\nDeeper layers represent larger subtree structures; shallow features sometimes generalize better, so all are retained.\n\nContinue the code above and compare two graph pairs using randomly weighted GIN-0, with a two-layer MLP in each layer:\n\n```python\nrng = np.random.default_rng(0)\nmlps = [(rng.normal(size=(4, 8)), rng.normal(size=8), rng.normal(size=(8, 4)), rng.normal(size=4))\n        for _ in range(3)]                            # One two-layer MLP per layer, including biases\n\ndef to_adj(g):\n    n, edges = g\n    M = np.zeros((n, n))\n    for i, j in edges:\n        M[i, j] = M[j, i] = 1\n    return M\n\ndef gin(g, eps=0.0):\n    M = to_adj(g)\n    h = np.ones((len(M), 4))                          # No node features: everyone starts with the same vector\n    readouts = [h.sum(0)]\n    for W1, b1, W2, b2 in mlps:\n        m = (1 + eps) * h + M @ h                     # (1+epsilon) * self + sum of neighbors\n        h = np.maximum(np.maximum(m @ W1 + b1, 0) @ W2 + b2, 0)   # Two-layer MLP\n        readouts.append(h.sum(0))                     # Sum readout at every layer\n    return np.concatenate(readouts)                   # Concatenate layers into a graph representation\n\nhexagon = (6, [(0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 0)])\ntwo_triangles = (6, [(0, 1), (1, 2), (2, 0), (3, 4), (4, 5), (5, 3)])\nfor name, g1, g2 in [(\"Path vs star\", path, star), (\"Hexagon vs two triangles\", hexagon, two_triangles)]:\n    print(name, \"Same representation:\", np.allclose(gin(g1), gin(g2)))\n```\n\n```text\nPath vs star Same representation: False\nHexagon vs two triangles Same representation: True\n```\n\nThe path and star separate. The hexagon and two triangles do not, whatever weights we choose: every node in both graphs has exactly two neighbors, so every WL round sees the same thing. GIN must likewise return identical representations. One graph has two triangles and the other none, yet an aggregation-based GNN cannot see the difference.\n\n\n> [!TIP]\n> \n> Looking only at the first-layer sum readout, removing the MLP biases makes the path and star indistinguishable. A bias-free ReLU network satisfies $f(cx)=cf(x)$ for $c>0$. With all-one inputs, the first layer's final sum reduces to node count plus total degree; both graphs have 4 nodes and total degree 6. Lemma 7 discusses this kind of failure: a single-layer perceptron without bias maps some different multisets to the same result.\n\n\n### Weaker Can Still Be Useful\n\nThe authors also place GCN and GraphSAGE in this framework. With single-layer perceptrons, mean or max, they are weaker than WL but still useful. Mean learns distributions; when a task cares about statistical proportions of neighboring features rather than exact structure, it suffices. With diverse, rarely repeated features, mean is as powerful as sum. The authors suggest this explains why mean-based models such as GCN work well on feature-rich tasks like paper-topic classification and community detection.\n\nMax learns the underlying set, suiting tasks that need representative elements.\n\n## Discussion\n\n### Experimental Setup\n\nThere are 9 graph-classification datasets: 4 from bioinformatics (MUTAG, PTC, NCI1, PROTEINS) and 5 social-network datasets (COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, REDDIT-MULTI5K). Bioinformatics nodes have categorical features. Social-network nodes lack features: REDDIT uses the same number for every node, while the others use one-hot degrees. Evaluation uses 10-fold cross-validation.\n\nComparisons include graph kernels, existing GNNs and GIN variants replacing sum with mean or max, or MLP with a single-layer perceptron. GCN corresponds to mean/single-layer, GraphSAGE to max/single-layer.\n\n### Can It Fit the Training Set?\n\nThe theory concerns expressiveness, so training accuracy is the most direct test. Training curves show MLP versions fitting better than single-layer perceptrons, and sum better than mean or max. GIN-0 and GIN-$\\epsilon$ fit almost every training set to nearly 100%.\n\nNo GNN exceeds the WL subtree kernel's training accuracy, consistent with the WL ceiling.\n\n### Test Accuracy\n\n| Dataset | IMDB-B | IMDB-M | RDT-B | RDT-M5K | COLLAB | MUTAG | PROTEINS | PTC | NCI1 |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| WL subtree | 73.8 | 50.9 | 81.0 | 52.5 | 78.9 | **90.4** | 75.0 | 59.9 | **86.0** |\n| **GIN-0** | **75.1** | **52.3** | **92.4** | **57.5** | **80.2** | 89.4 | **76.2** | 64.6 | 82.7 |\n| Mean, MLP | 73.7 | 52.3 | 50.0 | 20.0 | 79.2 | 83.5 | 75.5 | **66.6** | 80.9 |\n| Mean, single-layer (GCN) | 74.0 | 51.9 | 50.0 | 20.0 | 79.0 | 85.6 | 76.0 | 64.2 | 80.2 |\n| Max, single-layer (GraphSAGE) | 72.3 | 50.9 | — | — | — | 85.1 | 75.9 | 63.9 | 77.7 |\n\nThis table shows selected methods, with mean 10-fold accuracy (%). The two REDDIT datasets stand out. All nodes have identical features, leaving mean-based models blind to structure. RDT-B binary accuracy is 50.0 ± 0.0%, and RDT-M5K five-class accuracy is 20.0 ± 0.0%—chance level. GIN-0 reaches 92.4% and 57.5%. Even giving degrees as features, the authors add, brings mean-based models only to 71.2% and 41.3%.\n\nGIN is best or tied best on all five social-network datasets. Bioinformatics results are less one-sided. On NCI1, the WL subtree kernel clearly wins (86.0% versus 82.7%); on MUTAG, GIN trails WL and PATCHY-SAN's 92.6%; on PTC, mean/MLP wins. These datasets are small and standard deviations large: GIN-0 on PTC scores 64.6 ± 7.0%.\n\nGIN-0 and GIN-$\\epsilon$ fit training data similarly, but GIN-0 consistently does slightly better on tests, perhaps because it is simpler.\n\n## Conclusion\n\nGIN turns GNN power into a provable question. Neighbor-aggregation GNNs can distinguish graphs at most as well as WL. Reaching that ceiling requires multiset-injective aggregation: sum can do this, mean and max cannot. Designed accordingly, GIN uses sum and an MLP to substantially outperform mean-based models on social-network graph classification.\n\nThe ceiling remains. GIN can never distinguish a hexagon from two triangles; later work adds substructure counts, positional encodings or higher-order WL to get past this wall.\n\nFor graph classification with few node features, aggregate by sum, and remember the MLP biases.\n","imageSizes":{"/content/papers/assets/1062d104bcaae50e5f8c9abf6c72f574a014c68e6a7bb4c6705f663336773b4f.png":{"width":960,"height":214},"/content/papers/assets/33c4b052156c190bc85033ddb47e50c70b500425938ac89108383fc039b88837.png":{"width":1108,"height":240}}}