{"notes":[{"id":"graph-learning/appnp","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"APPNP","subtitle":"Remember the Way Home","author":"Z. Yuan","paperDate":"2018-10-14","publishedOn":"2024-01-17","world":"external/appnp","paperTitle":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","paperUrl":"https://arxiv.org/abs/1810.05997","bibliography":{"title":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","authors":["Johannes Gasteiger","Aleksandar Bojchevski","Stephan Günnemann"],"url":"https://arxiv.org/abs/1810.05997","pdfUrl":"https://arxiv.org/pdf/1810.05997v6","codeUrl":"https://www.kdd.in.tum.de/ppnp","preprint":{"id":"1810.05997","submittedOn":"2018-10-14","url":"https://arxiv.org/abs/1810.05997"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.05997"},"citation":"@misc{graphlearningappnp2018,\n  title = {{Predict then Propagate: Graph Neural Networks meet Personalized PageRank}},\n  author = {Johannes Gasteiger and Aleksandar Bojchevski and Stephan Günnemann},\n  year = {2018},\n  eprint = {1810.05997},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.05997}\n}\n","citationPath":"/content/papers/citations/graph-learning/appnp.eae593f21b4d.bib"},"thumbnail":{"src":"/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png","width":908,"height":370},"locale":"en","path":"/content/papers/en/graph-learning/appnp.e8692cf97cf4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-happens-at-infinite-depth","text":"What Happens at Infinite Depth?","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"personalized-pagerank","text":"Personalized PageRank","depth":3},{"id":"ppnp-predict-then-propagate","text":"PPNP: Predict, Then Propagate","depth":3},{"id":"appnp-approximate-with-power-iteration","text":"APPNP: Approximate with Power Iteration","depth":3},{"id":"how-it-differs-from-gcn","text":"How It Differs from GCN","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"rigorous-evaluation","text":"Rigorous Evaluation","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"how-many-steps-how-often-to-return","text":"How Many Steps, How Often to Return?","depth":3},{"id":"where-to-apply-propagation","text":"Where to Apply Propagation","depth":3},{"id":"with-very-few-labels","text":"With Very Few Labels","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""}]},{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"Smoothed Too Flat","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"en","path":"/content/papers/en/graph-learning/deeper-insights.ce733c5df590.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"compare-with-a-fully-connected-network-first","text":"Compare with a Fully Connected Network First","depth":3},{"id":"laplacian-smoothing","text":"Laplacian Smoothing","depth":3},{"id":"too-much-smoothing","text":"Too Much Smoothing","depth":3},{"id":"too-shallow-is-a-problem-too","text":"Too Shallow Is a Problem Too","depth":3},{"id":"a-remedy-give-gcn-more-labels-to-see","text":"A Remedy: Give GCN More Labels to See","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]}],"text":"In a two-layer GCN, each node sees only neighbors within two hops.\n\nThe intuitive way to see farther is to stack more layers. Yet the GCN paper itself shows accuracy falling beyond two or three. Greater depth makes every node's representation increasingly alike, until they cannot be distinguished.\n\nJohannes Gasteiger, Aleksandar Bojchevski and Stephan Günnemann at the Technical University of Munich approach this differently. What does GCN converge to with infinitely many layers? A distribution independent of the starting node. Then make it remember the start: at every step, give it a chance to return home.\n\n(The first author published as Johannes Klicpera; the 2022 arXiv version uses his current name.)\n\n## Defining the Problem\n\nThe authors give two reasons why message passing models such as GCN struggle to enlarge their neighborhoods.\n\nFirst, mean aggregation causes **over-smoothing** as layers accumulate, losing focus on the local neighborhood.\n\n\n> [!TIP]\n> \n> For the origin of over-smoothing, see our note:\n> \n> - **[[18.01] Deeper Insights: Smoothed Too Flat](/blog/papers/graph-learning/deeper-insights)**\n\n\nSecond, most aggregators give every layer its own weight matrix. Seeing farther requires a deeper network and more parameters. But required reach and required network depth are separate questions; tying them together forces poor compromises.\n\n### What Happens at Infinite Depth?\n\nFirst address the former problem. Citing the Jumping Knowledge analysis, the authors explain that in a $k$-layer GCN, the expected influence of node $x$ on node $y$ is proportional to the probability of a slightly modified $k$-step random walk from $x$ reaching $y$. As $k\\rightarrow\\infty$, an irreducible, aperiodic graph makes this converge to the walk's limiting distribution.\n\nThat limiting distribution is a property of the entire graph, independent of the starting node.\n\nTry a seven-node path. Ignore weights and nonlinearities and examine propagation alone. Column $j$ of $\\hat A^K$ gives the proportions in which node $j$'s features spread across nodes after $K$ layers:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\nn = 7\nA = np.zeros((n, n))\nfor i in range(n - 1):                      # A seven-node path: 0-1-2-3-4-5-6\n    A[i, i + 1] = A[i + 1, i] = 1\nA_t = A + np.eye(n)\nd = A_t.sum(1)\nA_hat = A_t / np.sqrt(np.outer(d, d))\n\nfor K in (2, 10, 100):\n    P = np.linalg.matrix_power(A_hat, K)    # K-layer GCN propagation, ignoring weights and nonlinearities\n    print(f\"K={K:>3}  From node 0: {P[:, 0]}  From node 6: {P[:, 6]}\")\n```\n\n```text\nK=  2  From node 0: [0.42 0.34 0.14 0.   0.   0.   0.  ]  From node 6: [0.   0.   0.   0.   0.14 0.34 0.42]\nK= 10  From node 0: [0.2  0.22 0.18 0.12 0.07 0.04 0.02]  From node 6: [0.02 0.04 0.07 0.12 0.18 0.22 0.2 ]\nK=100  From node 0: [0.11 0.13 0.13 0.13 0.13 0.13 0.11]  From node 6: [0.11 0.13 0.13 0.13 0.13 0.13 0.11]\n```\n\nAt two layers, each endpoint influences its own neighborhood. At ten, influence covers most of the path. After a hundred, the two distributions are identical; neither remembers where it came from.\n\n## Solving the Problem\n\n### Personalized PageRank\n\nThe random walk's limiting distribution is PageRank. A variant developed for search engines, **personalized PageRank**, teleports back to a designated root node $x$ with probability $\\alpha$ at each step.\n\n$$\n\\pi_{\\text{ppr}}(i_x)=(1-\\alpha)\\hat A\\,\\pi_{\\text{ppr}}(i_x)+\\alpha i_x\n$$\n\n$i_x$ is the root's indicator vector. Solving gives:\n\n$$\n\\pi_{\\text{ppr}}(i_x)=\\alpha\\left(I_n-(1-\\alpha)\\hat A\\right)^{-1}i_x\n$$\n\nHere $\\hat A$ is the symmetrically normalized adjacency matrix with self-loops, exactly as in GCN. Replace $i_x$ with the identity matrix to get the graph's personalized PageRank matrix:\n\n$$\n\\Pi_{\\text{ppr}}=\\alpha\\left(I_n-(1-\\alpha)\\hat A\\right)^{-1}\n$$\n\n$(\\Pi_{\\text{ppr}})_{yx}$ is $x$'s influence on $y$. The continuing chance to return home keeps even limiting influence concentrated around the root, diminishing with distance. $\\alpha$ controls how rapidly it decays.\n\nOn the same path, set $\\alpha=0.1$:\n\n```python\nalpha = 0.1\nPi = alpha * np.linalg.inv(np.eye(n) - (1 - alpha) * A_hat)   # Personalized PageRank matrix\nprint(\"From node 0:\", Pi[:, 0])\nprint(\"From node 6:\", Pi[:, 6])\n\nH = np.eye(n)                               # Use one-hot predictions for each node\nZ = H.copy()\nfor k in range(1, 31):\n    Z = (1 - alpha) * A_hat @ Z + alpha * H # One APPNP step\n    if k in (2, 10, 30):\n        print(f\"After {k:>2} steps, max difference from exact solution: {np.abs(Z - Pi).max():.3f}\")\n```\n\n```text\nFrom node 0: [0.34 0.24 0.13 0.08 0.05 0.03 0.02]\nFrom node 6: [0.02 0.03 0.05 0.08 0.13 0.24 0.34]\nAfter  2 steps, max difference from exact solution: 0.143\nAfter 10 steps, max difference from exact solution: 0.015\nAfter 30 steps, max difference from exact solution: 0.000\n```\n\nEven after infinitely many steps, the endpoints' distributions remain concentrated around themselves. Leave the second half of the output aside for now; the next section will use it.\n\n### PPNP: Predict, Then Propagate\n\nThe authors separate the model into two parts. First, neural network $f_\\theta$ predicts for each node **independently**, ignoring the graph. Second, personalized PageRank propagates those predictions:\n\n$$\nZ_{\\text{PPNP}}=\\mathrm{softmax}\\left(\\alpha\\left(I_n-(1-\\alpha)\\hat A\\right)^{-1}H\\right),\\qquad H_{i,:}=f_\\theta(X_{i,:})\n$$\n\nThis is PPNP: personalized propagation of neural predictions.\n\n[![Figure 1 of the paper: on the left, each node's features x_i independently pass through neural network f_θ to produce prediction h_i. On the right, personalized PageRank propagates along graph edges in green, while dashed red arrows mark teleportation back to the root with probability α at each step, producing z_i.](/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png)](/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png)\n\n*Figure 1 from Gasteiger et al., taken from [**PDF page 3**](https://arxiv.org/pdf/1810.05997v6#page=3). The original caption is omitted.*\n\nThis separation solves the second problem too. Propagation has no parameters; $\\alpha$ controls reach, independently of neural-network depth. $f_\\theta$ could even be another kind of model, such as an image CNN.\n\nInference predicts then propagates, but training remains end to end: gradients pass through propagation back to $f_\\theta$, implicitly accounting for infinitely many aggregation layers.\n\n### APPNP: Approximate with Power Iteration\n\nPPNP has a practical problem. $\\Pi_{\\text{ppr}}$ is a dense $n\\times n$ matrix. Storage and each propagation cost $O(n^2)$; computing the exact inverse adds a separate preprocessing cost.\n\nTreat each column of $H$, one class, as a topic, and this becomes topic-sensitive PageRank, approximated by power iteration. Each step is a random walk with restart:\n\n$$\nZ^{(0)}=H,\\qquad Z^{(k+1)}=(1-\\alpha)\\hat AZ^{(k)}+\\alpha H,\\qquad Z^{(K)}=\\mathrm{softmax}\\left((1-\\alpha)\\hat AZ^{(K-1)}+\\alpha H\\right)\n$$\n\n$H$ is both the starting point and the destination for every restart. Each step needs one sparse matrix multiplication, without constructing an $n\\times n$ matrix. This is APPNP.\n\nReturn to the second half of the earlier code. Replace predictions $H$ with one-hot vectors and iteration approximates $\\Pi_{\\text{ppr}}$ itself. At 10 steps, the maximum difference from the exact solution is only 0.015; after 30, no difference is visible at the printed precision.\n\n### How It Differs from GCN\n\nWith $\\alpha=0$, APPNP propagation becomes GCN-style $\\hat A^K$. The only difference is the term $\\alpha H$ added back at every step.\n\nParameters are another difference: GCN adds weights for every propagation step; APPNP keeps all parameters in $f_\\theta$, however many steps it runs.\n\n## Discussion\n\n### Rigorous Evaluation\n\nThe paper takes evaluation seriously. The authors criticize using a single split, unclear separation of validation and test sets, and tuning hyperparameters by dataset or even split, despite message passing models' sensitivity to splits and initialization.\n\nThey instead use:\n\n- 100 runs per experiment across multiple random splits and initializations.\n- A visible set and a test set, using the test set once at the end and never for hyperparameter or model selection.\n- Shared depth, hidden units, dropout, L2 and learning rate across datasets. Each model's hyperparameters are grid-searched on Citeseer and Cora-ML, with the same early-stopping rule.\n- Bootstrap 95% confidence intervals and paired t-test p-values for main conclusions.\n\nAll four datasets use only their largest connected component: citation networks Citeseer, Cora-ML and PubMed, and the MS Academic coauthorship graph. APPNP uses $\\alpha=0.1$, $K=10$; MS Academic's different structure calls for $\\alpha=0.2$.\n\n### Main Results\n\n| Method | Citeseer | Cora-ML | PubMed | MS Academic |\n| --- | ---: | ---: | ---: | ---: |\n| V. GCN | 73.51 | 82.30 | 77.65 | 91.65 |\n| GCN | 75.40 | 83.41 | 78.68 | 92.10 |\n| N-GCN | 74.25 | 82.25 | 77.43 | 92.86 |\n| GAT | 75.39 | 84.37 | 77.76 | 91.22 |\n| JK | 73.03 | 82.69 | 77.88 | 91.71 |\n| PPNP | **75.83** | **85.29** | Out of memory | Out of memory |\n| APPNP | 75.73 | 85.09 | **79.73** | **93.27** |\n\nValues are mean accuracies (%), with most confidence intervals within ±0.5. The table caption is direct: previously reported improvements vanish under this rigorous setup. GAT and GCN differ by only 0.01 percentage points on Citeseer. PPNP and APPNP, meanwhile, win significantly across all four datasets.\n\nAPPNP takes about 25% longer per epoch than GCN on average because of more matrix multiplications, but is far faster than GAT: on PubMed, 64.1 ms versus 1029.8 ms.\n\n### How Many Steps, How Often to Return?\n\n[![Figure 4 of the paper: four dataset panels plot accuracy against propagation steps K (0, 10, 20 and infinity). Blue GCN-style propagation (α=0) loses accuracy sharply as K rises, while orange APPNP (α=0.1) improves then levels off.](/content/papers/assets/541a2a6626ac4eda1d710c61856129ff289cfdff224f1b0bd1a6a894c4877c13.png)](/content/papers/assets/541a2a6626ac4eda1d710c61856129ff289cfdff224f1b0bd1a6a894c4877c13.png)\n\n*Figure 4 from Gasteiger et al., taken from [**PDF page 7**](https://arxiv.org/pdf/1810.05997v6#page=7). The original caption is omitted.*\n\nBlue points use $\\alpha=0$, GCN-style propagation, whose accuracy collapses with more steps as it approaches global PageRank. Orange points use $\\alpha=0.1$: accuracy rises and converges to infinite-step PPNP. The authors find $K=10$ close enough, coinciding with the largest shortest-path distance from any dataset node to the training set.\n\nThe best teleportation probability $\\alpha$ varies by dataset, roughly from 0.05 to 0.2. Larger $\\alpha$ converges faster.\n\n### Where to Apply Propagation\n\nAn ablation compares propagation in training only, inference only, both or neither. Neither gives an MLP using node features alone.\n\nBoth is best. Interestingly, inference-only propagation also works quite well: train an ordinary MLP, then spread its predictions over the graph at inference. For an already trained model, this is an improvement at almost no additional cost.\n\n### With Very Few Labels\n\nThe fewer the labels, the greater APPNP's advantage over GCN. The authors explain that wider propagation carries information from scarce training nodes farther. Appendix analysis likewise shows that nodes farther from the training set benefit more from increased reach.\n\n## Conclusion\n\nStarting from GCN's relationship to random walks, APPNP replaces PageRank that forgets its starting point with personalized PageRank that returns home. More propagation steps avoid over-smoothing, propagation adds no parameters, and network depth can finally be chosen independently of reach.\n\nThe evaluation is as memorable as the model. The same GCN and GAT, assessed over multiple splits, 100 runs and a test set used only once, lose many previously reported gains.\n\nStart with the paper's $\\alpha=0.1$ and $K=10$. If you already have a trained MLP, try propagating its predictions only at inference, without retraining, and see how many percentage points it gains.\n","imageSizes":{"/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png":{"width":908,"height":370},"/content/papers/assets/541a2a6626ac4eda1d710c61856129ff289cfdff224f1b0bd1a6a894c4877c13.png":{"width":1068,"height":274}}}