{"notes":[{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"Smoothed Too Flat","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"en","path":"/content/papers/en/graph-learning/deeper-insights.ce733c5df590.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"compare-with-a-fully-connected-network-first","text":"Compare with a Fully Connected Network First","depth":3},{"id":"laplacian-smoothing","text":"Laplacian Smoothing","depth":3},{"id":"too-much-smoothing","text":"Too Much Smoothing","depth":3},{"id":"too-shallow-is-a-problem-too","text":"Too Shallow Is a Problem Too","depth":3},{"id":"a-remedy-give-gcn-more-labels-to-see","text":"A Remedy: Give GCN More Labels to See","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/appnp","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"APPNP","subtitle":"Remember the Way Home","author":"Z. Yuan","paperDate":"2018-10-14","publishedOn":"2024-01-17","world":"external/appnp","paperTitle":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","paperUrl":"https://arxiv.org/abs/1810.05997","bibliography":{"title":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","authors":["Johannes Gasteiger","Aleksandar Bojchevski","Stephan Günnemann"],"url":"https://arxiv.org/abs/1810.05997","pdfUrl":"https://arxiv.org/pdf/1810.05997v6","codeUrl":"https://www.kdd.in.tum.de/ppnp","preprint":{"id":"1810.05997","submittedOn":"2018-10-14","url":"https://arxiv.org/abs/1810.05997"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.05997"},"citation":"@misc{graphlearningappnp2018,\n  title = {{Predict then Propagate: Graph Neural Networks meet Personalized PageRank}},\n  author = {Johannes Gasteiger and Aleksandar Bojchevski and Stephan Günnemann},\n  year = {2018},\n  eprint = {1810.05997},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.05997}\n}\n","citationPath":"/content/papers/citations/graph-learning/appnp.eae593f21b4d.bib"},"thumbnail":{"src":"/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png","width":908,"height":370},"locale":"en","path":"/content/papers/en/graph-learning/appnp.e8692cf97cf4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-happens-at-infinite-depth","text":"What Happens at Infinite Depth?","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"personalized-pagerank","text":"Personalized PageRank","depth":3},{"id":"ppnp-predict-then-propagate","text":"PPNP: Predict, Then Propagate","depth":3},{"id":"appnp-approximate-with-power-iteration","text":"APPNP: Approximate with Power Iteration","depth":3},{"id":"how-it-differs-from-gcn","text":"How It Differs from GCN","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"rigorous-evaluation","text":"Rigorous Evaluation","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"how-many-steps-how-often-to-return","text":"How Many Steps, How Often to Return?","depth":3},{"id":"where-to-apply-propagation","text":"Where to Apply Propagation","depth":3},{"id":"with-very-few-labels","text":"With Very Few Labels","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""}]},{"id":"graph-learning/sgc","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SGC","subtitle":"Is Linear Enough?","author":"Z. Yuan","paperDate":"2019-02-19","publishedOn":"2024-02-09","world":"external/sgc","paperTitle":"Simplifying Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1902.07153","bibliography":{"title":"Simplifying Graph Convolutional Networks","authors":["Felix Wu","Tianyi Zhang","Amauri Holanda de Souza","Christopher Fifty","Tao Yu","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1902.07153","pdfUrl":"https://arxiv.org/pdf/1902.07153v2","codeUrl":"https://github.com/Tiiiger/SGC","preprint":{"id":"1902.07153","submittedOn":"2019-02-19","url":"https://arxiv.org/abs/1902.07153"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/wu19e.html"},"citation":"@misc{graphlearningsgc2019,\n  title = {{Simplifying Graph Convolutional Networks}},\n  author = {Felix Wu and Tianyi Zhang and Amauri Holanda de Souza and Christopher Fifty and Tao Yu and Kilian Q. Weinberger},\n  year = {2019},\n  eprint = {1902.07153},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.07153}\n}\n","citationPath":"/content/papers/citations/graph-learning/sgc.5b3815dbf17c.bib"},"thumbnail":{"src":"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png","width":1366,"height":612},"locale":"en","path":"/content/papers/en/graph-learning/sgc.b9d6518d0273.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"remove-the-nonlinearities","text":"Remove the Nonlinearities","depth":3},{"id":"a-spectral-view-a-low-pass-filter","text":"A Spectral View: A Low-Pass Filter","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"citation-networks-and-reddit","text":"Citation Networks and Reddit","depth":3},{"id":"how-much-faster","text":"How Much Faster?","depth":3},{"id":"downstream-tasks","text":"Downstream Tasks","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gin","relation":"referenced","topic":""}]}],"text":"When GCN appears in late 2016, it takes a substantial lead in semi-supervised classification on three citation networks. Yet the paper leaves its effectiveness somewhat unexplained: starting from spectral graph convolution, a succession of approximations ends in one short propagation rule.\n\nQimai Li, Zhichao Han and Xiao-Ming Wu at Hong Kong Polytechnic University want to unpack that line. What is it doing? When does it fail?\n\nTheir answer is that GCN's graph convolution is Laplacian smoothing. A little smoothing helps; too much makes all nodes alike. This is **over-smoothing**, and later papers such as APPNP and over-squashing cite this work when discussing why GNNs cannot go deep.\n\n## Defining the Problem\n\nThe authors identify two practical problems with GCN.\n\nFirst, it is shallow: the original paper uses two layers. A two-layer GCN sees only two-hop neighbors. With very few labels, most nodes have no labeled node nearby, so label information cannot reach the whole graph.\n\nSecond, GCN depends on a validation set. The original paper trains with 20 labels per class—140 on Cora—yet uses another 500 labeled nodes for validation and early stopping. A validation set larger than the training set conflicts with the purpose of semi-supervised learning and makes comparisons with methods requiring no validation set less fair.\n\nThe authors first seek to understand why GCN works, then propose improvements for both problems.\n\n## Solving the Problem\n\n### Compare with a Fully Connected Network First\n\nThe only difference between GCN and a fully connected network (FCN) is the extra left multiplication by $\\hat A=\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}$. On Cora with 20 labels per class:\n\n| One-layer FCN | Two-layer FCN | One-layer GCN | Two-layer GCN |\n| ---: | ---: | ---: | ---: |\n| 53.1% | 55.9% | 70.8% | 79.8% |\n\nEven one GCN layer gains nearly 18 percentage points over FCN. The entire difference is $\\hat A$.\n\n### Laplacian Smoothing\n\nSplit one GCN layer into two steps: graph convolution $Y=\\hat AX$, followed by a fully connected layer. The key is clearly the first step.\n\nLaplacian smoothing is an old mesh-smoothing method in graphics. For each feature channel, every node mixes its value with those of its neighbors:\n\n$$\n\\hat y_i=(1-\\gamma)x_i+\\gamma\\sum_j\\frac{\\tilde a_{ij}}{d_i}x_j,\\qquad 0<\\gamma\\le1\n$$\n\nIn matrix form, $\\hat Y=(I-\\gamma\\tilde D^{-1}\\tilde L)X$, where $\\tilde L=\\tilde D-\\tilde A$. Setting $\\gamma=1$ gives the neighbor average, including self, $\\tilde D^{-1}\\tilde AX$. Replacing $\\tilde D^{-1}\\tilde L$ with symmetrically normalized $\\tilde D^{-1/2}\\tilde L\\tilde D^{-1/2}$ gives exactly GCN's $\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}X$.\n\nGCN's graph convolution is therefore **symmetric Laplacian smoothing**. Each node's features become a weighted average of its own and its neighbors'. On citation networks, connected nodes tend to share a class; averaging makes same-class nodes more alike, easing the classifier's job. The authors see this as the key to GCN's effectiveness.\n\n### Too Much Smoothing\n\nIf smoothing helps, why not stack more layers?\n\nThe authors use Zachary's karate club: 34 members and 78 edges, eventually splitting into two factions. GCN weights are randomly initialized without training, node features are one-hot, the hidden dimension is 16 and the output dimension 2. They plot outputs for 1 to 5 layers:\n\n[![Figure 2 of the paper: five scatterplots show two-dimensional embeddings from untrained GCNs with 1 to 5 layers. Yellow and black dots represent the factions. At 1 layer the colors mix; at 2 they mostly separate; from 3 to 5 they mix again, with the 5-layer points squeezed into a thin line.](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)\n\n*Figure 2 from Li et al., taken from [**PDF page 4**](https://arxiv.org/pdf/1801.07606v1#page=4). The main caption is omitted.*\n\nOne layer separates poorly; two largely separate the factions; more layers mix them again. The authors explain that the dataset is small and has many cross-faction edges, so mixing happens especially quickly.\n\nRemove random weights and examine smoothing alone. Smooth each node's one-hot features $k$ times, then compare cosine similarities within and between factions:\n\n```python\nimport networkx as nx\nimport numpy as np\n\nG = nx.karate_club_graph()\nA = nx.to_numpy_array(G, weight=None) + np.eye(34)        # 34 members, 78 edges, with self-loops\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nclub = np.array([G.nodes[v][\"club\"] == \"Mr. Hi\" for v in G])   # The club later split into two factions\nsame = club[:, None] == club[None, :]\n\nfor k in (1, 2, 4, 16, 64):\n    H = np.linalg.matrix_power(A_hat, k)                   # Identity features: H = Â^k I\n    Hn = H / np.linalg.norm(H, axis=1, keepdims=True)\n    S = Hn @ Hn.T                                          # Pairwise cosine similarity\n    print(f\"k={k:>2}  same faction {S[same].mean():.3f}   different factions {S[~same].mean():.3f}\")\n```\n\n```text\nk= 1  same faction 0.273   different factions 0.054\nk= 2  same faction 0.542   different factions 0.179\nk= 4  same faction 0.818   different factions 0.424\nk=16  same faction 0.990   different factions 0.942\nk=64  same faction 1.000   different factions 1.000\n```\n\nAfter a few rounds, members of the same faction are indeed more alike than members of different factions: this is where GCN helps. But between-faction similarity also rises. At 64 rounds both are 1.000, everyone's features point the same way, and the factions can no longer be separated.\n\nThe authors formalize this as a theorem. If the graph has no bipartite connected component, repeated Laplacian smoothing makes features within each connected component converge to the same value; the symmetric version converges to values proportional to the square root of degree. A graph with self-loops is necessarily non-bipartite, so enough GCN layers inevitably reach this point.\n\n### Too Shallow Is a Problem Too\n\nIf depth fails, the network must stay shallow. But shallowness has its own cost.\n\n[![Figure 1 of the paper: the horizontal axis is the Cora training-set fraction from 0.01 to 0.05; the vertical axis is test accuracy. Label propagation in blue rises slowly from 0.62 to 0.70. GCN with validation in orange rises from around 0.62 to 0.79; without validation, the green GCN curve starts near 0.54 at 0.01. The authors' red curve is highest throughout, from around 0.70 to 0.82.](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)\n\n*Figure 1 from Li et al., taken from [**PDF page 2**](https://arxiv.org/pdf/1801.07606v1#page=2). The original caption is omitted.*\n\nAs labels decrease, GCN falls faster than label propagation, which uses only graph structure. The authors explain that two GCN layers pass labels only two hops and miss global structure, whereas label propagation spreads over the whole graph. Removing validation hurts GCN further.\n\n### A Remedy: Give GCN More Labels to See\n\nThe authors change the training data rather than the model, expanding GCN's label set.\n\n- **Co-training**: a random-walk model that sees the whole graph—partially absorbing random walks—finds the nodes closest to known labels in each class and adds them to training.\n- **Self-training**: train a GCN, add its most confident predictions to training, then continue. This is especially useful when many small isolated components prevent random walks from carrying information across.\n- **Union** and **Intersection**: take the union or intersection of the labels found by both methods. Union gives more diverse labels; intersection keeps only those both trust and is more accurate.\n\nAll four train directly on the expanded labels without an additional validation set.\n\n## Discussion\n\nThe authors reduce label rates on Cora, CiteSeer and PubMed far below the original GCN settings. For Cora:\n\n| Label rate | 0.5% | 1% | 2% | 3% | 4% | 5% |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| Label propagation | 56.4 | 62.3 | 65.4 | 67.5 | 69.0 | 70.2 |\n| ChebNet | 38.0 | 52.0 | 62.4 | 70.8 | 74.1 | 77.6 |\n| GCN, without validation | 42.6 | 56.9 | 67.8 | 74.9 | 77.6 | 79.3 |\n| GCN, with validation | 50.9 | 62.3 | 72.2 | 76.5 | 78.4 | 79.7 |\n| Union | **58.5** | **69.9** | **75.9** | **78.5** | **80.4** | **81.7** |\n\nValues are accuracies (%). At 0.5% labels, GCN without validation reaches only 42.6%, below label propagation's 56.4%; Union reaches 58.5%. At 1% labels on Cora and CiteSeer, the authors report gains of 23% and 28% over GCN without validation; at 0.05% labels on PubMed, the gain is 37%.\n\nWith enough labels, the gap disappears: the supplied labels already suffice to train a good GCN. ChebNet performs poorly in most settings, which the authors attribute to possible overfitting.\n\nAnother striking result is that Union, using only 2% of Cora's labels (54 nodes), beats every baseline in the original GCN table trained on 140 labels.\n\n## Conclusion\n\nThis paper explains GCN's graph convolution as symmetric Laplacian smoothing. Smoothing makes same-class nodes more alike, which helps GCN; too much makes all nodes converge to a common pattern determined only by degree, causing over-smoothing. They are two sides of the same operation.\n\nGCN must therefore remain shallow, yet a shallow GCN cannot pass sparse labels far enough. Random walks and self-training expand the labels and open a clear gap in extremely low-label settings.\n\nMore than two years later, Alon and Yahav write in their over-squashing paper that over-smoothing is the accepted explanation for GNNs' limited depth. APPNP later in the same year as this paper, and SGC the following year, take two paths: one tries to propagate without blurring away information; the other keeps only the smoothing.\n\n\n> [!TIP]\n> \n> For these two later paths, see our notes:\n> \n> - **[[18.10] APPNP: Remember the Way Home](/blog/papers/graph-learning/appnp)**\n> - **[[19.02] SGC: Is Linear Enough?](/blog/papers/graph-learning/sgc)**\n\n\nNext time your GCN worsens with depth, do not rush to blame the optimizer. Extract the intermediate features and compute pairwise similarities; they may already have been smoothed flat.\n","imageSizes":{"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png":{"width":1396,"height":268},"/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png":{"width":628,"height":302}}}