{"notes":[{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"Is One Parameter Enough?","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"en","path":"/content/papers/en/graph-learning/gcn.61f7394131fa.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-graph-as-a-regularizer","text":"The Graph as a Regularizer","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"simplifying-from-the-spectral-domain","text":"Simplifying from the Spectral Domain","depth":3},{"id":"one-gcn-layer","text":"One GCN Layer","depth":3},{"id":"two-layers-are-enough","text":"Two Layers Are Enough","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-propagation-rule-works-best","text":"Which Propagation Rule Works Best?","depth":3},{"id":"what-happens-with-more-layers","text":"What Happens with More Layers?","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"a-review-four-days-later","text":"A Review Four Days Later","depth":3},{"id":"the-papers-own-limitations","text":"The Paper's Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]},{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"A Detour through the Spectral Domain","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"en","path":"/content/papers/en/graph-learning/spectral-networks.cc53a07db581.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"spatial-construction-local-connections-and-clustering","text":"Spatial Construction: Local Connections and Clustering","depth":3},{"id":"spectral-construction-laplacian-eigenvectors","text":"Spectral Construction: Laplacian Eigenvectors","depth":3},{"id":"rediscovering-cnns","text":"Rediscovering CNNs","depth":3},{"id":"smooth-coefficients-local-filters","text":"Smooth Coefficients, Local Filters","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"subsampled-mnist","text":"Subsampled MNIST","depth":3},{"id":"mnist-on-a-sphere","text":"MNIST on a Sphere","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"A Polynomial Will Do","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"en","path":"/content/papers/en/graph-learning/chebnet.6f3f6320bdd0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"translation-has-no-spatial-definition","text":"Translation Has No Spatial Definition","depth":3},{"id":"spectral-computation-is-too-expensive","text":"Spectral Computation Is Too Expensive","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"polynomials-buy-locality","text":"Polynomials Buy Locality","depth":3},{"id":"chebyshev-recursion-buys-speed","text":"Chebyshev Recursion Buys Speed","depth":3},{"id":"pooling-on-graphs","text":"Pooling on Graphs","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"first-validate-on-a-grid","text":"First Validate on a Grid","depth":3},{"id":"comparing-three-filters","text":"Comparing Three Filters","depth":3},{"id":"how-good-is-the-graph","text":"How Good Is the Graph?","depth":3},{"id":"what-came-next","text":"What Came Next","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]}],"text":"You have 2,708 papers, each with bag-of-words features, linked by 5,429 citations. Only 140 have topic labels—20 for each of seven topics. The model must guess the rest.\n\nThis is semi-supervised node classification on Cora. Intuitively, papers that cite one another should concern the same topic; the question is how to put that intuition into a model.\n\nThomas Kipf and Max Welling at the University of Amsterdam give a very short answer. At each layer, every node takes a weighted average of its own and its neighbors' features using fixed proportions, then multiplies by a weight matrix. Stack two layers and stop. On Cora, this model reaches 81.5%, against the previous best of 75.7%.\n\nIt is called GCN, a name that later becomes almost synonymous with graph neural networks.\n\n## Defining the Problem\n\n### The Graph as a Regularizer\n\nBefore GCN, a common approach put the graph into the loss:\n\n$$\n\\mathcal{L}=\\mathcal{L}_0+\\lambda\\,\\mathcal{L}_{\\text{reg}},\\qquad\n\\mathcal{L}_{\\text{reg}}=\\sum_{i,j}A_{ij}\\left\\|f(X_i)-f(X_j)\\right\\|^2\n$$\n\n$\\mathcal{L}_0$ is the supervised loss on labeled nodes, while $\\mathcal{L}_{\\text{reg}}$ penalizes large differences in connected nodes' outputs. Label propagation and manifold regularization belong to this family.\n\nThe authors point out that these methods assume an edge signifies similarity. Many citations fit that assumption, but edges may carry other information, too. Hard-coding them as similarity restricts what the model can learn.\n\nAnother approach, graph embedding such as DeepWalk, first runs random walks to learn a vector for each node, then trains a classifier. These steps are optimized separately; a poorly tuned earlier step cannot be rescued by the final classifier.\n\nThe authors want a model $f(X, A)$ that takes the graph directly. $X$ contains node features and $A$ is the adjacency matrix. Loss is computed only on labeled nodes; gradients flow through $A$ to unlabeled nodes, learning their representations together.\n\n## Solving the Problem\n\n### Simplifying from the Spectral Domain\n\nThe derivation starts with spectral graph convolution. The normalized Laplacian $L=I_N-D^{-1/2}AD^{-1/2}$ decomposes into $U\\Lambda U^\\top$, with each column of $U$ representing a graph “frequency.” Convolving a node signal $x$ transforms it to the spectral domain, multiplies by a filter, and transforms back:\n\n$$\ng_\\theta\\star x=U g_\\theta U^\\top x\n$$\n\nThis is expensive. One multiplication by $U$ costs $O(N^2)$, and eigendecomposition alone is prohibitive on large graphs. ChebNet replaces the filter with a Chebyshev polynomial: an order-$K$ approximation avoids eigendecomposition and sees only neighbors within $K$ hops.\n\n\n> [!TIP]\n> \n> For the origins of spectral graph convolution and ChebNet's polynomial filters, see our notes:\n> \n> - **[[13.12] Spectral Networks: A Detour through the Spectral Domain](/blog/papers/graph-learning/spectral-networks)**\n> - **[[16.06] ChebNet: A Polynomial Will Do](/blog/papers/graph-learning/chebnet)**\n\n\nKipf and Welling make three further cuts.\n\nFirst, keep only first order, $K=1$. A layer sees one hop; more layers extend its reach. The authors argue that social and citation networks have broad degree distributions, so more flexible filters are more prone to overfitting local structures.\n\nSecond, approximate the largest eigenvalue $\\lambda_{\\max}$ as 2 and let the network parameters adapt to the scale difference. The filter becomes:\n\n$$\ng_{\\theta'}\\star x\\approx\\theta'_0x-\\theta'_1D^{-\\frac12}AD^{-\\frac12}x\n$$\n\nThird, tie the two parameters together, $\\theta=\\theta'_0=-\\theta'_1$:\n\n$$\ng_\\theta\\star x\\approx\\theta\\left(I_N+D^{-\\frac12}AD^{-\\frac12}\\right)x\n$$\n\nThe matrix in parentheses has eigenvalues in $[0,2]$. With many layers, the largest value, 2, repeatedly amplifies its component, potentially causing numerical instability and exploding or vanishing gradients. The authors apply a **renormalization trick**: add self-loops, $\\tilde A=A+I_N$, and normalize using the new degrees $\\tilde D_{ii}=\\sum_j\\tilde A_{ij}$:\n\n$$\nI_N+D^{-\\frac12}AD^{-\\frac12}\\;\\rightarrow\\;\\tilde D^{-\\frac12}\\tilde A\\tilde D^{-\\frac12}\n$$\n\nCompare the eigenvalues on a small five-node graph:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\n# Citations among five papers as an undirected graph: 0-1, 0-2, 1-2, 2-3, 3-4\nA = np.zeros((5, 5))\nfor i, j in [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]:\n    A[i, j] = A[j, i] = 1\n\nd = A.sum(1)\nS = np.eye(5) + A / np.sqrt(np.outer(d, d))     # Eq. 7: I + D^-1/2 A D^-1/2\nA_t = A + np.eye(5)                             # Add self-loops\nd_t = A_t.sum(1)\nA_hat = A_t / np.sqrt(np.outer(d_t, d_t))       # Renormalization: D̃^-1/2 à D̃^-1/2\n\nprint(\"Eq. 7 eigenvalues:\", np.linalg.eigvalsh(S))\nprint(\" eigenvalues:   \", np.linalg.eigvalsh(A_hat))\n```\n\n```text\nEq. 7 eigenvalues: [0.14 0.5  0.7  1.65 2.  ]\n eigenvalues:    [-0.21  0.    0.17  0.79  1.  ]\n```\n\nEquation 7 has largest eigenvalue 2, so ten applications amplify its component over a thousandfold. The eigenvalues of $\\hat A$ stay at most 1, making stacking numerically manageable.\n\n### One GCN Layer\n\nGeneralize a single signal to $C$-dimensional features per node, $X\\in\\mathbb{R}^{N\\times C}$, and apply $F$ filters:\n\n$$\nZ=\\hat AX\\Theta,\\qquad \\hat A=\\tilde D^{-\\frac12}\\tilde A\\tilde D^{-\\frac12}\n$$\n\n$\\Theta\\in\\mathbb{R}^{C\\times F}$ contains learnable weights. Add nonlinearities and stack layers to obtain Equation 2 of the paper:\n\n$$\nH^{(l+1)}=\\sigma\\left(\\hat AH^{(l)}W^{(l)}\\right)\n$$\n\nAt the node level, $\\hat A_{ij}=1/\\sqrt{\\tilde d_i\\tilde d_j}$. The share node $i$ receives from neighbor $j$ depends only on their degrees, and all neighbors share the same $W$. $\\hat AX$ can be computed as a sparse–dense matrix product, costing $O(|E|FC)$ per layer, linear in the edge count.\n\n### Two Layers Are Enough\n\nFor semi-supervised classification, the paper stacks just two layers:\n\n$$\nZ=f(X,A)=\\mathrm{softmax}\\left(\\hat A\\,\\mathrm{ReLU}\\left(\\hat AXW^{(0)}\\right)W^{(1)}\\right)\n$$\n\n$\\hat A$ is precomputed once. The loss is cross-entropy, computed only on labeled nodes.\n\n[![Figure 1 of the paper: the left panel diagrams a multilayer GCN. The input has C channels, nodes X1 to X4 connect along graph edges, and hidden layers produce F feature maps; some nodes have labels Y. The right panel is a t-SNE of a two-layer GCN's hidden outputs on Cora, with colors representing topics that form separate clusters.](/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png)](/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png)\n\n*Figure 1 from Kipf and Welling, taken from [**PDF page 4**](https://arxiv.org/pdf/1609.02907v4#page=4). The original caption is omitted.*\n\nThe left panel emphasizes that every layer shares the same graph; only features change. On the right, training with only 5% of Cora's labels already gives hidden outputs whose t-SNE separates the seven colors into clusters.\n\nContinuing with the small graph above, run a two-layer GCN forward pass. Each node has 4 features, the hidden layer has 3 dimensions, and there are 2 classes; only nodes 0 and 4 are labeled:\n\n```python\nrng = np.random.default_rng(0)\nX = rng.random((5, 4))            # 5 nodes, 4 features each (C = 4)\nW0 = rng.normal(size=(4, 3))      # Input to hidden layer (H = 3)\nW1 = rng.normal(size=(3, 2))      # Hidden layer to 2 classes (F = 2)\n\nH1 = np.maximum(A_hat @ X @ W0, 0)                     # ReLU(ÂXW0)\nlogits = A_hat @ H1 @ W1                               #  H W1\nZ = np.exp(logits) / np.exp(logits).sum(1, keepdims=True)\nprint(\"H1:\", H1.shape, \" Z:\", Z.shape)\nprint(Z)\n\nlabeled, y = [0, 4], np.array([0, 1])                  # Only nodes 0 and 4 have labels\nprint(\"loss:\", -np.log(Z[labeled, y]).mean().round(3))\n```\n\n```text\nH1: (5, 3)  Z: (5, 2)\n[[0.47 0.53]\n [0.47 0.53]\n [0.47 0.53]\n [0.47 0.53]\n [0.48 0.52]]\nloss: 0.7\n```\n\nThe weights are untrained, so every node assigns probabilities near 0.5 to the two classes, and the loss is roughly $\\ln 2$. During training, only losses at nodes 0 and 4 propagate back, but $\\hat A$ links them to their neighbors, updating the representations of nodes 1, 2 and 3 as well.\n\nThe paper uses full-batch gradient descent over the whole graph, keeping all of $\\hat A$ and $X$ in memory at every step. Dropout is the only source of stochasticity.\n\n## Discussion\n\n### Experimental Setup\n\nThe experiments follow Planetoid's setup. The citation networks Citeseer, Cora and Pubmed have 3,327, 2,708 and 19,717 nodes, respectively, with only 20 labels per class. The NELL knowledge graph has 65,755 nodes and only 1 label per class. A 500-node validation set is used only for hyperparameter tuning, and the test set has 1,000 nodes.\n\nThe model is a two-layer GCN with a 16-dimensional hidden layer, dropout 0.5, first-layer L2 regularization $5\\times10^{-4}$, and Adam with learning rate 0.01. Training runs for at most 200 epochs, stopping when validation loss fails to improve for 10 consecutive epochs. Hyperparameters are tuned only on Cora, then transferred unchanged to Citeseer and Pubmed.\n\n### Main Results\n\n| Method | Citeseer | Cora | Pubmed | NELL |\n| --- | ---: | ---: | ---: | ---: |\n| ManiReg | 60.1 | 59.5 | 70.7 | 21.8 |\n| LP | 45.3 | 68.0 | 63.0 | 26.5 |\n| DeepWalk | 43.2 | 67.2 | 65.3 | 58.1 |\n| ICA | 69.1 | 75.1 | 73.9 | 23.1 |\n| Planetoid* | 64.7 | 75.7 | 77.2 | 61.9 |\n| **GCN** | **70.3** | **81.5** | **79.0** | **66.0** |\n| GCN (random splits) | 67.9 ± 0.5 | 80.1 ± 0.5 | 78.9 ± 0.7 | 58.4 ± 1.7 |\n\nThe values are classification accuracies (%); the GCN row averages 100 random initializations. On the same machine, GCN converges faster than Planetoid on Citeseer, Cora and NELL (7, 4 and 48 seconds versus 26, 13 and 185), while Pubmed takes 38 seconds versus 25.\n\nThe last row uses 10 random splits of the same size. Accuracy falls somewhat, most sharply on NELL, from 66.0% to 58.4%.\n\n### Which Propagation Rule Works Best?\n\nThe authors replace the two layers' propagation with the different versions in the derivation, keeping everything else fixed:\n\n| Propagation rule | Citeseer | Cora | Pubmed |\n| --- | ---: | ---: | ---: |\n| Chebyshev, $K=3$ | 69.8 | 79.5 | 74.4 |\n| Chebyshev, $K=2$ | 69.6 | 81.2 | 73.8 |\n| First order, two parameters (Eq. 6) | 68.3 | 80.0 | 77.5 |\n| First order, one parameter (Eq. 7) | 69.3 | 79.2 | 77.4 |\n| **Renormalization (Eq. 8)** | **70.3** | **81.5** | **79.0** |\n| First-order term only | 68.7 | 80.5 | 77.8 |\n| MLP (without the graph) | 46.5 | 55.1 | 71.4 |\n\nFurther simplification does not hurt the results. Renormalization uses the fewest parameters and operations and wins on all three datasets; higher-order Chebyshev filters fall to around 74% on Pubmed. The graph-free MLP reaches only 55.1% on Cora, making the graph structure's contribution clear.\n\n### What Happens with More Layers?\n\nAppendix B varies the depth from 1 to 10 layers, uses all labels in 5-fold cross-validation, and compares models with and without residual connections:\n\n[![Figure 5 of the paper: the three panels show Citeseer, Cora and Pubmed, with 1 to 10 layers on the horizontal axis and accuracy on the vertical axis. Training and test curves peak around 2 to 3 layers. Dashed curves without residuals fall sharply beyond 7 layers; solid curves with residuals remain roughly level.](/content/papers/assets/8678c3fe0abb2cce0863dc23cf622ad78187872d1ddebf951b3a6a7385d15405.png)](/content/papers/assets/8678c3fe0abb2cce0863dc23cf622ad78187872d1ddebf951b3a6a7385d15405.png)\n\n*Figure 5 from Kipf and Welling, taken from [**PDF page 14**](https://arxiv.org/pdf/1609.02907v4#page=14). The original caption is omitted.*\n\nAll three datasets work best at 2 to 3 layers. Beyond 7 layers, even training accuracy drops without residuals; the authors say training becomes difficult. Residuals keep the curves roughly level but bring no further improvement.\n\n### The Connection to WL\n\nAppendix A offers another interpretation of GCN. Weisfeiler–Lehman (WL) is a graph-isomorphism test that updates each node's color by hashing its own color and the multiset of neighboring colors. Replace the hash with a learnable neural layer and choose normalization $\\sqrt{d_id_j}$, and this becomes GCN's propagation rule. The authors describe GCN as a differentiable, parameterized generalization of WL, adding their own qualifier, “loosely speaking.”\n\nFollowing this analogy, they experiment with the 34 members of Zachary's karate club. With three GCN layers, entirely random untrained weights and identity-matrix node features, the two-dimensional output embeddings already resemble the community structure. The authors compare this with DeepWalk results that require training. Giving one label per community and training for 300 steps makes the four communities linearly separable.\n\n### A Review Four Days Later\n\nFour days after the arXiv upload, Ferenc Huszár publishes a blog review titled “How powerful are Graph Convolutions?” He checks the model on an image grid with 8-neighbor connectivity and finds that every layer and channel uses the same fixed-shape 3×3 kernel, differing only by a scalar multiplier. He argues that such a model cannot learn edge detectors or Gabor filters; the simplifications greatly reduce its generality, while ChebNet's higher-order polynomials look more promising. [inFERENCe, 2016-09-13](https://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/)\n\nWe checked the final $\\hat A$: in the grid interior, the central pixel assigns 1/9 to each of the surrounding 9 positions, including itself. It is an averaging filter.\n\nKipf responds in a blog post at the end of the month. He agrees that degenerating to simple operations on regular grids is a shared drawback of models for arbitrary graphs; they should be evaluated beyond regular grids, with their tradeoffs acknowledged. [Kipf, 2016-09-30](https://tkipf.github.io/graph-convolutional-networks/)\n\nBoth have a point. On images, GCN loses CNNs' ability to assign weights by position; on directionless citation graphs with widely varying degrees, the propagation comparison above shows that simplification works better.\n\n### The Paper's Own Limitations\n\n- **Memory**: full-batch training memory grows linearly with the data. With minibatches, a $K$-layer model must load all neighbors within $K$ hops of each node.\n- **Edge types**: the model supports only undirected graphs and no edge features. NELL's directed, labeled edges must first be converted into a bipartite graph.\n- **Self-loop weight**: the model implicitly treats the node itself as equally important as its neighbors. The authors suggest $\\tilde A=A+\\lambda I_N$, making $\\lambda$ learnable.\n\nSampling methods such as GraphSAGE later address the first point. GraphSAGE also offers another approach to the third, weighting the node itself and its neighbors separately.\n\n## Conclusion\n\nStarting from spectral graph convolution, GCN cuts down to first order and one parameter, then stabilizes the numbers with renormalization. The result is a propagation rule that fits on one line. Two layers substantially beat the methods of the day on three citation networks and train quickly.\n\nThe costs are equally clear. Neighbor weights depend only on degrees, leaving just an averaging kernel on an image grid; performance starts to deteriorate beyond two or three layers. Many later ideas in GraphSAGE, GAT and GIN start from these two points.\n\nTo try it yourself, PyTorch Geometric's `GCNConv` implements this propagation rule. Follow the paper with two layers, 16 hidden dimensions and dropout 0.5; the paper reports convergence on Cora in just 4 seconds.\n\nStart with two layers. More is not always better.\n","imageSizes":{"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png":{"width":1122,"height":400},"/content/papers/assets/8678c3fe0abb2cce0863dc23cf622ad78187872d1ddebf951b3a6a7385d15405.png":{"width":1132,"height":454}}}