{"notes":[{"id":"graph-learning/sgc","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SGC","subtitle":"Is Linear Enough?","author":"Z. Yuan","paperDate":"2019-02-19","publishedOn":"2024-02-09","world":"external/sgc","paperTitle":"Simplifying Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1902.07153","bibliography":{"title":"Simplifying Graph Convolutional Networks","authors":["Felix Wu","Tianyi Zhang","Amauri Holanda de Souza","Christopher Fifty","Tao Yu","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1902.07153","pdfUrl":"https://arxiv.org/pdf/1902.07153v2","codeUrl":"https://github.com/Tiiiger/SGC","preprint":{"id":"1902.07153","submittedOn":"2019-02-19","url":"https://arxiv.org/abs/1902.07153"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/wu19e.html"},"citation":"@misc{graphlearningsgc2019,\n  title = {{Simplifying Graph Convolutional Networks}},\n  author = {Felix Wu and Tianyi Zhang and Amauri Holanda de Souza and Christopher Fifty and Tao Yu and Kilian Q. Weinberger},\n  year = {2019},\n  eprint = {1902.07153},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.07153}\n}\n","citationPath":"/content/papers/citations/graph-learning/sgc.5b3815dbf17c.bib"},"thumbnail":{"src":"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png","width":1366,"height":612},"locale":"en","path":"/content/papers/en/graph-learning/sgc.b9d6518d0273.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"remove-the-nonlinearities","text":"Remove the Nonlinearities","depth":3},{"id":"a-spectral-view-a-low-pass-filter","text":"A Spectral View: A Low-Pass Filter","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"citation-networks-and-reddit","text":"Citation Networks and Reddit","depth":3},{"id":"how-much-faster","text":"How Much Faster?","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gin","relation":"referenced","topic":""}]},{"id":"graph-learning/gin","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GIN","subtitle":"Summing Is What Counts","author":"Z. Yuan","paperDate":"2018-10-01","publishedOn":"2023-12-28","world":"external/gin","paperTitle":"How Powerful are Graph Neural Networks?","paperUrl":"https://arxiv.org/abs/1810.00826","bibliography":{"title":"How Powerful are Graph Neural Networks?","authors":["Keyulu Xu","Weihua Hu","Jure Leskovec","Stefanie Jegelka"],"url":"https://arxiv.org/abs/1810.00826","pdfUrl":"https://arxiv.org/pdf/1810.00826v3","codeUrl":"https://github.com/weihua916/powerful-gnns","preprint":{"id":"1810.00826","submittedOn":"2018-10-01","url":"https://arxiv.org/abs/1810.00826"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.00826"},"citation":"@misc{graphlearninggin2018,\n  title = {{How Powerful are Graph Neural Networks?}},\n  author = {Keyulu Xu and Weihua Hu and Jure Leskovec and Stefanie Jegelka},\n  year = {2018},\n  eprint = {1810.00826},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.00826}\n}\n","citationPath":"/content/papers/citations/graph-learning/gin.f2b31912b576.bib"},"thumbnail":{"src":"/content/papers/assets/1062d104bcaae50e5f8c9abf6c72f574a014c68e6a7bb4c6705f663336773b4f.png","width":960,"height":214},"locale":"en","path":"/content/papers/en/graph-learning/gin.8710e40a579a.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"a-common-form-for-neighbor-aggregation","text":"A Common Form for Neighbor Aggregation","depth":3},{"id":"neighbors-form-a-multiset","text":"Neighbors Form a Multiset","depth":3},{"id":"the-wl-test","text":"The WL Test","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-ceiling-for-gnns","text":"The Ceiling for GNNs","depth":3},{"id":"sum-mean-and-max","text":"Sum, Mean and Max","depth":3},{"id":"gin","text":"GIN","depth":3},{"id":"weaker-can-still-be-useful","text":"Weaker Can Still Be Useful","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"can-it-fit-the-training-set","text":"Can It Fit the Training Set?","depth":3},{"id":"test-accuracy","text":"Test Accuracy","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/sgc","relation":"citing","topic":""},{"id":"graph-learning/gnn-plus","relation":"citing","topic":""}]}],"text":"GCN borrows a whole package from deep learning: layer after layer, each with a weight matrix, ReLUs in between and softmax at the end.\n\nFelix Wu, Tianyi Zhang, Kilian Weinberger and colleagues at Cornell suspect some of this may be unnecessary. Their introduction recalls that machine learning usually starts simple and adds complexity when needed: linear perceptrons' limitations led to multilayer perceptrons; fixed filters such as Sobel later evolved into learnable CNNs. GCN is a rare exception, appearing after the neural-network revival and starting directly with multilayer networks, without a simpler linear version.\n\nThey add a practical observation: most classifiers in real applications are still linear, usually logistic regression, easy to optimize and interpret.\n\nSo they strip GCN down to its simplest form and see what remains.\n\n## Defining the Problem\n\nRecall one GCN layer. Let $S=\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}$ be the normalized adjacency with self-loops. Layer $k$ does three things:\n\n1. **Feature propagation**: $\\bar H^{(k)}\\leftarrow SH^{(k-1)}$, a weighted average with neighbors.\n2. **Linear transformation**: $\\bar H^{(k)}\\leftarrow\\bar H^{(k)}\\Theta^{(k)}$.\n3. **Nonlinearity**: $H^{(k)}\\leftarrow\\mathrm{ReLU}(\\bar H^{(k)})$.\n\nAfter $K$ layers, softmax classifies:\n\n$$\n\\hat Y_{\\text{GCN}}=\\mathrm{softmax}\\left(SH^{(K-1)}\\Theta^{(K)}\\right)\n$$\n\nWhich of these three really makes GCN work on citation networks? The authors hypothesize the first. Adjacent nodes in these graphs usually share a class; averaging neighboring features makes same-class nodes more alike and simplifies classification.\n\n## Solving the Problem\n\n### Remove the Nonlinearities\n\nIf nonlinearities are not essential, remove them all, keeping only the final softmax:\n\n$$\n\\hat Y=\\mathrm{softmax}\\left(S\\cdots SSX\\Theta^{(1)}\\Theta^{(2)}\\cdots\\Theta^{(K)}\\right)\n$$\n\nRepeated $S$ becomes $S^K$, and the product of weight matrices collapses into one $\\Theta$:\n\n$$\n\\hat Y_{\\text{SGC}}=\\mathrm{softmax}\\left(S^KX\\Theta\\right)\n$$\n\nThis is SGC, Simple Graph Convolution. $\\bar X=S^KX$ has no parameters and can be precomputed once. Training then becomes multiclass logistic regression on $\\bar X$, with a convex loss, suitable for second-order methods or stochastic gradient descent.\n\n[![Figure 1 of the paper: the upper GCN repeatedly applies feature propagation, linear transformation and nonlinearity K times, then softmax. The lower SGC applies K-step propagation once at the front, followed by logistic regression. Both rows show the same input and predictions.](/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png)](/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png)\n\n*Figure 1 from Wu et al., taken from [**PDF page 2**](https://arxiv.org/pdf/1902.07153v2#page=2). The original caption is omitted.*\n\nGCN returns to the graph at every layer. SGC completes propagation at the start; the classifier afterward need not know what the graph looks like.\n\nTry a six-node path, 0-1-2-3-4-5, with the first three nodes in one class and the last three in another:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\nn = 6\nA = np.zeros((n, n))\nfor i in range(n - 1):                      # A six-node path\n    A[i, i + 1] = A[i + 1, i] = 1\n\ndef sym_norm(M):\n    d = M.sum(1)\n    return M / np.sqrt(np.outer(d, d))\n\nrng = np.random.default_rng(0)\nX = rng.random((n, 4))                      # 6 nodes, 4 features each\nS = sym_norm(A + np.eye(n))\nX_bar = np.linalg.matrix_power(S, 2) @ X    # K = 2: precompute; no graph access needed afterward\nprint(\"X̄:\", X_bar.shape)\n\n# The rest is ordinary multiclass logistic regression: softmax(X̄ Θ)\nTheta = np.zeros((4, 2))\ny = np.array([0, 0, 0, 1, 1, 1])\nfor _ in range(500):\n    logits = X_bar @ Theta\n    P = np.exp(logits - logits.max(1, keepdims=True))\n    P /= P.sum(1, keepdims=True)\n    Theta -= 0.5 * X_bar.T @ (P - np.eye(2)[y]) / n      # Cross-entropy gradient\nprint(\"Predictions:\", P.argmax(1), \" Labels:\", y)\n```\n\n```text\nX̄: (6, 4)\nPredictions: [0 0 0 1 1 1]  Labels: [0 0 0 1 1 1]\n```\n\nThis only demonstrates the procedure; six training points tell us little about accuracy. The point is that $S$ has vanished from the training loop. The graph is used only at the beginning.\n\n### A Spectral View: A Low-Pass Filter\n\nWhy should smoothing suffice? The authors offer a spectral explanation.\n\nGCN's derivation starts with the graph Laplacian $\\Delta_{\\text{sym}}=I-D^{-1/2}AD^{-1/2}$, whose eigenvalues $\\lambda$ are graph frequencies in $[0, 2]$. Without self-loops, propagation is $S_{\\text{adj}}=D^{-1/2}AD^{-1/2}=I-\\Delta_{\\text{sym}}$. Applying it $K$ times gives the frequency filter:\n\n$$\ng(\\lambda)=(1-\\lambda)^K\n$$\n\nHigh frequencies near $\\lambda=2$ are troublesome: $1-\\lambda$ is near $-1$, making coefficients negative for odd $K$ and positive for even $K$, without suppressing their magnitude.\n\nTheorem 1 proves that adding self-loops reduces the largest Laplacian eigenvalue while leaving the minimum at 0. On Cora, the maximum falls from roughly 2 to 1.5. $(1-\\lambda)^K$ stays in a smaller range across the spectrum, turning $S^K$ into a low-pass-type filter. This also explains GCN's renormalization trick.\n\nReturn to the path. It is bipartite, so without self-loops its maximum frequency is exactly 2:\n\n```python\nlam = np.linalg.eigvalsh(np.eye(n) - sym_norm(A)).clip(0)     # Frequencies corresponding to S_adj\nlam_t = np.linalg.eigvalsh(np.eye(n) - sym_norm(A + np.eye(n))).clip(0)  # After adding self-loops\nprint(\"Without self-loops:\", lam)\nprint(\"With self-loops:\", lam_t)\nfor K in (1, 2, 3):\n    print(f\"K={K}  Highest-frequency coefficient  Without self-loops: {(1 - lam.max()) ** K:5.2f}\"\n          f\"   With self-loops: {(1 - lam_t.max()) ** K:5.2f}\")\n```\n\n```text\nWithout self-loops: [0.   0.19 0.69 1.31 1.81 2.  ]\nWith self-loops: [0.   0.11 0.41 0.78 1.09 1.28]\nK=1  Highest-frequency coefficient  Without self-loops: -1.00   With self-loops: -0.28\nK=2  Highest-frequency coefficient  Without self-loops:  1.00   With self-loops:  0.08\nK=3  Highest-frequency coefficient  Without self-loops: -1.00   With self-loops: -0.02\n```\n\nWithout self-loops, the highest-frequency component keeps magnitude 1 however many times we multiply, merely flipping sign. With self-loops, the maximum frequency falls to 1.28, leaving only 0.08 after two applications.\n\n[![Figure 2 of the paper: three panels show a first-order Chebyshev filter, normalized adjacency without self-loops, and normalized adjacency with self-loops. The horizontal axis is eigenvalue 0 to 2; blue curves are filter coefficients for K=1 to 6, and red is the spectrum of Cora's third feature. First-order Chebyshev coefficients grow above 8 at low frequencies; without self-loops, high-frequency coefficients near 2 are ±1; with self-loops, the spectrum ends around 1.5 and coefficients decrease with frequency.](/content/papers/assets/8025ac636ebdc23923398f9d8902317850a9937978d6dc393b96103e730fce8d.png)](/content/papers/assets/8025ac636ebdc23923398f9d8902317850a9937978d6dc393b96103e730fce8d.png)\n\n*Figure 2 from Wu et al., taken from [**PDF page 5**](https://arxiv.org/pdf/1902.07153v2#page=5). The original caption is omitted.*\n\nEach panel represents a propagation matrix. On the left, the first-order Chebyshev filter from GCN's derivation, $I+D^{-1/2}AD^{-1/2}$, has coefficients $(2-\\lambda)^K$, exploding at low frequencies as $K$ rises—the instability noted in GCN. In the middle, without self-loops, high-frequency coefficients jump between $+1$ and $-1$. On the right, self-loops compress the spectrum to around 1.5, with a downward curve giving clean low-pass filtering.\n\nAppendix experiments echo this picture. On validation data, propagation with self-loops is more accurate and stable across $K$ values; without them, odd $K$ causes a clear accuracy drop.\n\n## Discussion\n\n### Citation Networks and Reddit\n\n| Method | Cora | Citeseer | Pubmed |\n| --- | ---: | ---: | ---: |\n| GCN | 81.4 ± 0.4 | 70.9 ± 0.5 | 79.0 ± 0.4 |\n| GAT | 83.3 ± 0.7 | 72.6 ± 0.6 | 78.5 ± 0.3 |\n| FastGCN | 79.8 ± 0.3 | 68.8 ± 0.6 | 77.4 ± 0.3 |\n| GIN | 77.6 ± 1.1 | 66.1 ± 0.9 | 77.0 ± 1.2 |\n| **SGC** | 81.0 ± 0.0 | **71.9 ± 0.1** | 78.9 ± 0.0 |\n\nThese are the authors' reruns, averaging accuracy (%) over 10 trials; SGC uses $K=2$. It almost ties GCN and gains roughly 1 percentage point on Citeseer, perhaps because fewer parameters reduce overfitting. GAT remains best on Cora and Citeseer. GIN, implemented from its paper and separately tuned, performs worse here through overfitting.\n\nSGC's standard deviation is almost zero. Its convex objective makes different initializations converge to nearly identical solutions.\n\nReddit has 230,000 nodes, exhausting memory for full GCN. SGC reaches micro-F1 94.9, ahead of sampling-based SAGE-GCN (93.0) and FastGCN (93.7), but below SAGE-mean (95.0), SAGE-LSTM (95.4) and GaAN (96.4). With unregularized L-BFGS, it converges in two steps.\n\n### How Much Faster?\n\nThe authors measure training time on a GTX 1080 Ti relative to SGC, including SGC's precomputation of $S^KX$. On Pubmed, GCN takes 28 times as long and FastGCN 6 times. On Reddit, SAGE-GCN takes 32 times, SAGE-LSTM 180 times and FastGCN 100 times.\n\n### Downstream Tasks\n\nThe authors substitute SGC for GCN in five downstream tasks:\n\n1. **Text classification**: TextGCN treats documents and words as nodes. SGC with $K=2$ performs comparably on 5 datasets, with up to 83.6× speedup.\n2. **User geolocation**: it beats GCN with highway connections on GEOTEXT, TWITTER-US and TWITTER-WORLD, saving over 30 hours on TWITTER-WORLD.\n3. **Relation extraction**: replacing C-GCN's GCN gives 67.0% accuracy on TACRED, versus 66.4% for C-GCN.\n4. **Zero-shot image classification**: MLP plus SGC replaces GCNZ with 55% fewer parameters and slightly higher accuracy.\n5. **Graph classification**: here it fails. Substituting into DCGCN gives 71.0% on NCI1 and 76.2% on COLLAB, close to GCN but far behind GIN. On the QM8 quantum-chemistry dataset, SGC has MAE 0.03 versus 0.01 for more advanced methods.\n\nIn the first four tasks, adjacent graph elements tend to be similar, so smoothing works. Graph classification distinguishes structure itself, where smoothing alone is insufficient.\n\n\n> [!TIP]\n> \n> For why GIN beats mean-based models on graph classification, see our note:\n> \n> - **[[18.10] GIN: Summing Is What Counts](/blog/papers/graph-learning/gin)**\n\n\n## Conclusion\n\nSGC reduces GCN to two steps: smooth over the graph $K$ times, then run logistic regression. On citation networks and many downstream tasks, it matches GCN while training one or two orders of magnitude faster. Spectrally, self-loops compress the spectrum and make $S^K$ a low-pass filter, supplying another explanation for GCN's renormalization trick.\n\nThese results also mark GCN's boundaries. On citation networks, its useful part is largely smoothing. When the task needs structure itself, as in graph classification, SGC and GCN fall behind together.\n\nWhen someone compares a new GCN variant, include SGC as a baseline. It trains in seconds; if the new model cannot beat it, the nonlinearities need a better case.\n","imageSizes":{"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png":{"width":1366,"height":612},"/content/papers/assets/8025ac636ebdc23923398f9d8902317850a9937978d6dc393b96103e730fce8d.png":{"width":1080,"height":396}}}