{"notes":[{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"抹得太平了","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/deeper-insights.3dfb15278376.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"先和全连接网络比","text":"先和全连接网络比","depth":3},{"id":"拉普拉斯平滑","text":"拉普拉斯平滑","depth":3},{"id":"平滑太多次","text":"平滑太多次","depth":3},{"id":"太浅也不行","text":"太浅也不行","depth":3},{"id":"补救让-gcn-多看到一些标签","text":"补救：让 GCN 多看到一些标签","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/appnp","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"APPNP","subtitle":"记得回家","author":"Z. Yuan","paperDate":"2018-10-14","publishedOn":"2024-01-17","world":"external/appnp","paperTitle":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","paperUrl":"https://arxiv.org/abs/1810.05997","bibliography":{"title":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","authors":["Johannes Gasteiger","Aleksandar Bojchevski","Stephan Günnemann"],"url":"https://arxiv.org/abs/1810.05997","pdfUrl":"https://arxiv.org/pdf/1810.05997v6","codeUrl":"https://www.kdd.in.tum.de/ppnp","preprint":{"id":"1810.05997","submittedOn":"2018-10-14","url":"https://arxiv.org/abs/1810.05997"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.05997"},"citation":"@misc{graphlearningappnp2018,\n  title = {{Predict then Propagate: Graph Neural Networks meet Personalized PageRank}},\n  author = {Johannes Gasteiger and Aleksandar Bojchevski and Stephan Günnemann},\n  year = {2018},\n  eprint = {1810.05997},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.05997}\n}\n","citationPath":"/content/papers/citations/graph-learning/appnp.eae593f21b4d.bib"},"thumbnail":{"src":"/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png","width":908,"height":370},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/appnp.84d419b594d8.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"叠到无限多层会怎样","text":"叠到无限多层会怎样","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"personalized-pagerank","text":"Personalized PageRank","depth":3},{"id":"ppnp先预测再传播","text":"PPNP：先预测，再传播","depth":3},{"id":"appnp用幂迭代近似","text":"APPNP：用幂迭代近似","depth":3},{"id":"和-gcn-差在哪","text":"和 GCN 差在哪","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"严格的评估","text":"严格的评估","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"要传几步多常回家","text":"要传几步、多常回家","depth":3},{"id":"传播用在哪里","text":"传播用在哪里","depth":3},{"id":"标签很少的时候","text":"标签很少的时候","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""}]},{"id":"graph-learning/sgc","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SGC","subtitle":"线性就够了？","author":"Z. Yuan","paperDate":"2019-02-19","publishedOn":"2024-02-09","world":"external/sgc","paperTitle":"Simplifying Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1902.07153","bibliography":{"title":"Simplifying Graph Convolutional Networks","authors":["Felix Wu","Tianyi Zhang","Amauri Holanda de Souza","Christopher Fifty","Tao Yu","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1902.07153","pdfUrl":"https://arxiv.org/pdf/1902.07153v2","codeUrl":"https://github.com/Tiiiger/SGC","preprint":{"id":"1902.07153","submittedOn":"2019-02-19","url":"https://arxiv.org/abs/1902.07153"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/wu19e.html"},"citation":"@misc{graphlearningsgc2019,\n  title = {{Simplifying Graph Convolutional Networks}},\n  author = {Felix Wu and Tianyi Zhang and Amauri Holanda de Souza and Christopher Fifty and Tao Yu and Kilian Q. Weinberger},\n  year = {2019},\n  eprint = {1902.07153},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.07153}\n}\n","citationPath":"/content/papers/citations/graph-learning/sgc.5b3815dbf17c.bib"},"thumbnail":{"src":"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png","width":1366,"height":612},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/sgc.fffc749dcc50.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"拿掉非线性","text":"拿掉非线性","depth":3},{"id":"从频域看它是一个低通滤波器","text":"从频域看：它是一个低通滤波器","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"引用网络与-reddit","text":"引用网络与 Reddit","depth":3},{"id":"快多少","text":"快多少","depth":3},{"id":"下游任务","text":"下游任务","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gin","relation":"referenced","topic":""}]}],"text":"GCN 在 2016 年底一推出，就在三个引用网络的半监督分类上大幅领先。但它为什么有效，论文本身没有说得很清楚；推导从频域图卷积出发，一路近似，最后得到一行很短的传播规则。\n\n香港理工大学的 Qimai Li、Zhichao Han 与 Xiao-Ming Wu 想把这一行公式拆开来看。它到底在做什么？什么情况下会失灵？\n\n他们的答案是，GCN 的图卷积是一种拉普拉斯平滑；平滑一次有用，平滑太多次，所有节点都会变得一样。这个现象叫做 **over-smoothing**，后来 APPNP、over-squashing 等论文谈到 GNN 叠不深时，引用的就是这篇。\n\n## 定义问题\n\n作者点出 GCN 的两个实际问题。\n\n第一，GCN 是浅层的，论文用的是两层。两层 GCN 只看得到两步以内的邻居；标签很少的时候，大部分节点身边根本没有任何有标签的节点，标签信息传不到整张图。\n\n第二，GCN 的训练依赖验证集。原论文每类只用 20 个标签训练，例如 Cora 一共 140 个，却另外用了 500 个有标签的节点做验证、决定何时停止训练。验证集比训练集还大，这违背了半监督学习的初衷，也让它和不需要验证集的方法比较时不太公平。\n\n作者想先弄清楚 GCN 有效的原因，再针对这两个问题提出改进。\n\n## 解决问题\n\n### 先和全连接网络比\n\nGCN 和全连接网络（FCN）唯一的差别，是在特征矩阵的左边多乘了一个 $\\hat A=\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}$。作者在 Cora 上，每类 20 个标签，比较两者：\n\n| 一层 FCN | 两层 FCN | 一层 GCN | 两层 GCN |\n| ---: | ---: | ---: | ---: |\n| 53.1% | 55.9% | 70.8% | 79.8% |\n\n只有一层时，GCN 就比 FCN 高了将近 18 个百分点。差别全在那个 $\\hat A$。\n\n### 拉普拉斯平滑\n\n把一层 GCN 拆成两步。先做图卷积 $Y=\\hat AX$，再接一层全连接。关键显然在第一步。\n\n拉普拉斯平滑是图形学里平滑网格的老方法。对每个特征通道，每个节点把自己和邻居的值混在一起：\n\n$$\n\\hat y_i=(1-\\gamma)x_i+\\gamma\\sum_j\\frac{\\tilde a_{ij}}{d_i}x_j,\\qquad 0<\\gamma\\le1\n$$\n\n写成矩阵是 $\\hat Y=(I-\\gamma\\tilde D^{-1}\\tilde L)X$，其中 $\\tilde L=\\tilde D-\\tilde A$。令 $\\gamma=1$，就是只用邻居（含自己）的平均 $\\tilde D^{-1}\\tilde AX$。再把 $\\tilde D^{-1}\\tilde L$ 换成对称归一化的 $\\tilde D^{-1/2}\\tilde L\\tilde D^{-1/2}$，得到的正好是 GCN 的 $\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}X$。\n\n所以 GCN 的图卷积是一种**对称拉普拉斯平滑**。它让每个节点的特征变成自己和邻居的加权平均；在引用网络这类图上，相连的节点倾向同一类，平均之后，同类的节点变得更像，后面的分类器就好做多了。作者认为，这就是 GCN 有效的关键。\n\n### 平滑太多次\n\n如果平滑是好事，多叠几层呢？\n\n作者用 Zachary 空手道俱乐部做实验。34 位成员、78 条边，后来分裂成两派。GCN 的权重随机初始化、不训练，节点特征用 one-hot，隐藏层 16 维，输出 2 维，画出 1 到 5 层的输出：\n\n[![原论文 Figure 2：五张散点图，分别是 1 到 5 层未训练 GCN 输出的二维嵌入，黄点与黑点代表两派成员。1 层时两色混在一起；2 层时大致分开；3 到 5 层时两色又逐渐混在一起，5 层时点都挤成一条细线。](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)\n\n*原论文 Figure 2（Li 等人），取自 [**PDF 第 4 页**](https://arxiv.org/pdf/1801.07606v1#page=4)。主图注未收录。*\n\n一层时分得不好，两层时两派大致分开，再往上叠，两派又混在一起。作者说，这个数据集很小，两派之间的连接也不少，所以混得特别快。\n\n我们可以把随机权重拿掉，只看平滑本身。把每个节点的 one-hot 特征平滑 $k$ 次，比较同一派和不同派成员之间的余弦相似度：\n\n```python\nimport networkx as nx\nimport numpy as np\n\nG = nx.karate_club_graph()\nA = nx.to_numpy_array(G, weight=None) + np.eye(34)        # 34 位成员、78 条边，加上自环\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nclub = np.array([G.nodes[v][\"club\"] == \"Mr. Hi\" for v in G])   # 俱乐部后来分裂成两派\nsame = club[:, None] == club[None, :]\n\nfor k in (1, 2, 4, 16, 64):\n    H = np.linalg.matrix_power(A_hat, k)                   # 特征用单位矩阵：H = Â^k I\n    Hn = H / np.linalg.norm(H, axis=1, keepdims=True)\n    S = Hn @ Hn.T                                          # 两两之间的余弦相似度\n    print(f\"k={k:>2}  同一派 {S[same].mean():.3f}   不同派 {S[~same].mean():.3f}\")\n```\n\n```text\nk= 1  同一派 0.273   不同派 0.054\nk= 2  同一派 0.542   不同派 0.179\nk= 4  同一派 0.818   不同派 0.424\nk=16  同一派 0.990   不同派 0.942\nk=64  同一派 1.000   不同派 1.000\n```\n\n平滑几次之后，同一派的成员确实比不同派的更像，这是 GCN 有用的地方。但不同派的相似度也一路上升，到了 64 次，两者都是 1.000，所有人的特征指向同一个方向，两派再也分不开。\n\n作者把这件事写成定理。只要图没有二部的连通分量，反复应用拉普拉斯平滑，每个连通分量里的节点特征都会收敛到同样的值；对称版本则收敛到和度数平方根成正比的值。加了自环的图一定不是二部图，所以 GCN 叠够多层一定会走到这一步。\n\n### 太浅也不行\n\n不能叠深，那就只好浅。但浅有浅的问题。\n\n[![原论文 Figure 1：横轴是 Cora 上的训练集比例，从 0.01 到 0.05，纵轴是测试准确率。标签传播的蓝线在 0.62 到 0.70 之间缓慢上升；有验证集的 GCN（橘线）从约 0.62 升到 0.79；没有验证集的 GCN（绿线）在 0.01 时只有约 0.54；作者方法的红线始终最高，从约 0.70 升到 0.82。](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)\n\n*原论文 Figure 1（Li 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/1801.07606v1#page=2)。图注未收录。*\n\n标签越少，GCN 掉得越快，比只用图结构的标签传播掉得还快。作者的解释是，两层 GCN 只能把标签信息传到两步以内，看不到整张图的结构；而标签传播本来就是沿着整张图扩散的。没有验证集时，GCN 掉得更多。\n\n### 补救：让 GCN 多看到一些标签\n\n作者的办法不改模型，改训练数据，替 GCN 扩充标签集。\n\n- **Co-training**：用一个能看到整张图的随机游走模型（partially absorbing random walks），替每一类找出离已知标签最近的几个节点，加进训练集。\n- **Self-training**：先训练一个 GCN，挑出它最有把握的预测，加进训练集，再继续训练。这在图有很多小的孤立分量、随机游走传不过去时特别有用。\n- **Union** 与 **Intersection**：把两种方法找到的标签取并集或交集。并集让标签更多样；交集只留两者都有把握的，比较准。\n\n这四种方法都直接在扩充后的标签集上训练，不需要额外的验证集。\n\n## 讨论\n\n作者在 Cora、CiteSeer、PubMed 上，把标签比例降到远低于原 GCN 论文的设置。以 Cora 为例：\n\n| 标签比例 | 0.5% | 1% | 2% | 3% | 4% | 5% |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| 标签传播 | 56.4 | 62.3 | 65.4 | 67.5 | 69.0 | 70.2 |\n| ChebNet | 38.0 | 52.0 | 62.4 | 70.8 | 74.1 | 77.6 |\n| GCN，无验证集 | 42.6 | 56.9 | 67.8 | 74.9 | 77.6 | 79.3 |\n| GCN，有验证集 | 50.9 | 62.3 | 72.2 | 76.5 | 78.4 | 79.7 |\n| Union | **58.5** | **69.9** | **75.9** | **78.5** | **80.4** | **81.7** |\n\n数字是准确率（%）。标签只有 0.5% 时，没有验证集的 GCN 只有 42.6%，连标签传播的 56.4% 都比不上；Union 是 58.5%。作者统计，在 Cora 和 CiteSeer 上标签比例 1% 时，他们的方法比没有验证集的 GCN 高 23% 和 28%；PubMed 上标签比例 0.05% 时，比没有验证集的 GCN 高 37%。\n\n标签够多时，差距就消失了，代表给定的标签已经足以训练出好的 GCN。ChebNet 在多数设置下表现不好，作者推测是过拟合。\n\n另一个值得一提的数字是，Union 在 Cora 上只用 2% 的标签（54 个），就胜过原 GCN 论文表中所有使用 140 个标签的基准方法。\n\n## 结论\n\n这篇论文把 GCN 的图卷积解释成对称拉普拉斯平滑。平滑让同类的节点更像，这是 GCN 有效的原因；平滑太多次，所有节点会收敛到只和度数有关的同一个样子，这就是 over-smoothing。两件事是同一个运算的两面。\n\nGCN 因此只能浅，而浅的 GCN 在标签很少时又传不远。作者用随机游走和自我训练替它扩充标签，在极少标签的设置下拉开了明显的差距。\n\n两年多后，Alon 与 Yahav 在讨论 over-squashing 时写道，GNN 叠不深的公认解释就是 over-smoothing。同一年稍晚的 APPNP 和隔年的 SGC，一个想办法让传播不会糊掉，一个干脆只留下平滑。\n\n\n> [!TIP]\n> \n> 这两条后续的路线，可以参考我们的笔记：\n> \n> - **[[18.10] APPNP：记得回家](/blog/papers/graph-learning/appnp)**\n> - **[[19.02] SGC：线性就够了？](/blog/papers/graph-learning/sgc)**\n\n\n下次你的 GCN 越叠越差，先别急着怪优化器。把中间层的特征拿出来算一下两两相似度，可能早就抹平了。\n","imageSizes":{"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png":{"width":1396,"height":268},"/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png":{"width":628,"height":302}}}