{"notes":[{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"塞不下了","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/over-squashing.1bfa8bae9091.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"问题半径","text":"问题半径","depth":3},{"id":"和-rnn-的瓶颈比较","text":"和 RNN 的瓶颈比较","depth":3},{"id":"只看传播比例","text":"只看传播比例","depth":3},{"id":"一个刻意设计的任务","text":"一个刻意设计的任务","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"全连接的最后一层","text":"全连接的最后一层","depth":3},{"id":"需要多大的向量","text":"需要多大的向量","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"生物数据集","text":"生物数据集","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"抹得太平了","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/deeper-insights.3dfb15278376.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"先和全连接网络比","text":"先和全连接网络比","depth":3},{"id":"拉普拉斯平滑","text":"拉普拉斯平滑","depth":3},{"id":"平滑太多次","text":"平滑太多次","depth":3},{"id":"太浅也不行","text":"太浅也不行","depth":3},{"id":"补救让-gcn-多看到一些标签","text":"补救：让 GCN 多看到一些标签","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"图上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphgps.74aedb083e17.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"位置与结构","text":"位置与结构","depth":3},{"id":"为什么消息传递需要它们","text":"为什么消息传递需要它们","depth":3},{"id":"先传递再注意会先丢掉信息","text":"先传递再注意，会先丢掉信息","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一层-gps","text":"一层 GPS","depth":3},{"id":"线性的复杂度","text":"线性的复杂度","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"消融两条路各自的份量","text":"消融：两条路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年后的重新评估","text":"一年后的重新评估","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"图神经网络从 2005 年前后被提出以来，就一直有个老问题，远方节点的信息很难传过来。\n\n实际应用中，GNN 叠超过几层就不再进步，大家接受的解释是 over-smoothing，层数一多，所有节点的表示都变得一样。Technion 的 Uri Alon 与 Eran Yahav 指出，over-smoothing 大多是在论文分类这类短距离任务上观察到的，这类任务的答案主要取决于附近的邻居，本来就不需要太多层。任务如果真的需要远方的信息，问题可能出在别的地方。\n\n\n> [!TIP]\n> \n> over-smoothing 的出处与它的证明，可以参考我们的笔记：\n> \n> - **[[18.01] Deeper Insights：抹得太平了](/blog/papers/graph-learning/deeper-insights)**\n\n\n他们注意到，要让距离 $r$ 的节点互相影响，至少得叠 $r$ 层；而每多叠一层，一个节点要照顾的邻居数量就呈指数成长，这些信息全都得塞进同一个固定长度的向量。他们把这个现象叫做 **over-squashing**，意思是信息被过度挤压。\n\n## 定义问题\n\n### 问题半径\n\n作者先定义**问题半径** $r$，也就是一个预测任务需要的交互距离。$r$ 通常事先不知道，实际应用中是靠调整层数 $K$ 来估计。\n\n节点 $v$ 在 $K$ 层之后的感受野 $\\mathcal N_v^K$，是它 $K$ 步以内的所有节点。如果任务需要的是长距离的交互，就必须 $K\\ge r$，否则远方的节点根本传不到，这叫做 **under-reaching**。\n\n但 $|\\mathcal N_v^K|$ 会随 $K$ 指数成长。在聚合那一步，这些指数成长的信息全被加总进一个固定长度的向量，关键的消息就送不到目的地。模型只好学会训练数据里的短距离信号，到了测试时就泛化得很差。\n\n### 和 RNN 的瓶颈比较\n\n这让人想起机器翻译在注意力出现之前的情况。seq2seq 的编码器得把整个输入句子压进最后一个隐藏状态，句子一长，信息就丢失了。\n\n[![原论文 Figure 1：左图是 RNN seq2seq 的瓶颈，输入序列的信息沿着一条链依序传到最后一个节点；右图是 GNN 的瓶颈，左侧呈树状展开的大量节点，信息一层一层汇集到最右边的目标节点，红色曲线在目标节点前挤成一束。](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)](/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png)\n\n*原论文 Figure 1（Alon 与 Yahav），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/2006.05205v4#page=2)。图注未收录。*\n\n作者认为 GNN 的瓶颈更严重。RNN 的感受野随步数**线性**成长，GNN 则是**指数**成长；同样是一个固定长度的向量，GNN 要塞的东西多得多。\n\n### 只看传播比例\n\n拿一棵二元树看看。只看传播的比例，不管权重和非线性，每个叶节点经过 $r$ 层之后，能在根节点里留下多少：\n\n```python\nimport numpy as np\n\ndef binary_tree(r):\n    n = 2 ** (r + 1) - 1                     # 根节点是 0，节点 i 的子节点是 2i+1、2i+2\n    A = np.zeros((n, n))\n    for i in range((n - 1) // 2):\n        for c in (2 * i + 1, 2 * i + 2):\n            A[i, c] = A[c, i] = 1\n    return A\n\nfor r in range(2, 8):\n    A = binary_tree(r) + np.eye(2 ** (r + 1) - 1)\n    d = A.sum(1)\n    A_hat = A / np.sqrt(np.outer(d, d))\n    share = np.linalg.matrix_power(A_hat, r)[0, -1]     # r 步之后，最后一个叶节点在根节点里的份量\n    print(f\"r={r}  叶节点 {2 ** r:>3} 个，每个叶节点在根节点里的份量 {share:.5f}\")\n```\n\n```text\nr=2  叶节点   4 个，每个叶节点在根节点里的份量 0.10206\nr=3  叶节点   8 个，每个叶节点在根节点里的份量 0.02552\nr=4  叶节点  16 个，每个叶节点在根节点里的份量 0.00638\nr=5  叶节点  32 个，每个叶节点在根节点里的份量 0.00159\nr=6  叶节点  64 个，每个叶节点在根节点里的份量 0.00040\nr=7  叶节点 128 个，每个叶节点在根节点里的份量 0.00010\n```\n\n每深一层，叶节点的数量加倍，每个叶节点在根节点里的份量却缩成大约四分之一。到了 $r=7$，128 个叶节点的信息混在一起，每个只占万分之一左右。这个示范只是直觉，论文的论证是从向量能存下多少信息出发。\n\n### 一个刻意设计的任务\n\n为了把瓶颈单独拿出来看，作者设计了 **NeighborsMatch** 任务。\n\n图里有几个绿色节点，每个绿色节点有不同数量的蓝色邻居，也各有一个字母标签。目标节点也有几个蓝色邻居，任务是预测它的标签，答案就是「蓝色邻居数量和它一样」的那个绿色节点的字母。每张图的对应关系都不同，所以模型每次都得把所有绿色节点的信息传到目标节点、逐一比对，没有捷径。\n\n这个任务看起来很刻意，但作者指出，它很像代码里的一类问题。一段 Python 程序声明了好几个变量，各有类型和值；之后某个位置该用哪个变量，得依照当下需要的类型和值，从可见的变量里挑。这就是后面会测的 VarMisuse 任务。\n\n**Tree-NeighborsMatch** 把中间那块结构换成一棵深度为 $r$ 的二元树。绿色节点都是叶子，目标节点是根，所有边都指向根。$r$ 就是问题半径，可以直接控制挤压的程度。模型用 $r+1$ 层，让信息传到根之后，还能再过一次非线性。\n\n## 解决问题\n\n这篇论文的主要贡献，是指出并解释 over-squashing；作者自己说，他们的修补方法只是最简单的一种，用来证明这个问题普遍存在、而且一直没被处理。\n\n### 全连接的最后一层\n\n做法是把最后一层改成**全连接层**（fully-adjacent，FA）。在这一层里，每一对节点都视为相邻，$\\mathcal N_v:=V$；层的种类不变，也不增加任何权重。\n\n前面 $K-1$ 层照原本的图做消息传递，学到带有结构信息的节点表示；最后一层让所有节点直接交互，信息不必再一层一层挤过瓶颈。\n\n### 需要多大的向量\n\n作者也算了一个组合下界，估计要在 Tree-NeighborsMatch 上达到 100% 训练准确率，隐藏维度 $d$ 至少要多大。下界随 $r$ 的阶乘成长，比指数还快；对 $d=32$（他们实验用的维度）来说，问题半径最多只能到 $r=7$。实际上更糟，梯度下降不保证找得到那个存在的解，实验里就算 $d=512$，也最多拟合到 $r=7$。\n\n## 讨论\n\n### Tree-NeighborsMatch\n\n[![原论文 Figure 3：横轴是问题半径 r 从 2 到 8，纵轴是训练准确率。GGNN、GAT、GIN、GCN 四条曲线在 r=2、3 都是 1.0；r=4 时 GGNN 和 GAT 仍为 1.0，GIN 掉到 0.77，GCN 掉到 0.70；之后全部下降，r=8 时都在 0.2 以下。](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)](/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png)\n\n*原论文 Figure 3（Alon 与 Yahav），取自 [**PDF 第 5 页**](https://arxiv.org/pdf/2006.05205v4#page=5)。图注未收录。*\n\n这张图看的是**训练**准确率，不是测试。GCN 和 GIN 只能完美拟合到 $r=3$，$r=4$ 时 GCN 只剩 70%；从 $r=5$ 开始，四种 GNN 都无法完美拟合训练集，即使每个训练样本已经看过上万次。\n\nGAT 和 GGNN 撑到 $r=4$，作者的解释在聚合的方式。GCN 和 GIN 先把所有邻居的消息加总成一个向量，之后才和目标节点自己的表示结合；GAT 用注意力，可以根据目标节点的表示挑选要看的邻居，只把相关的信息往上传。GGNN 也有类似的过滤，只是做在逐元素的闸门上。\n\n既然所有 GNN 在这里都拟合不了训练集，在公开的长距离数据集上，它们又是怎么拟合的？作者的假设是，模型靠的是训练集里的短距离信号与一些巧合的特征，没有真正学到被挤掉的长距离信息，所以测试时泛化不好。\n\n### QM9\n\nQM9 有约 13 万个分子，每个分子平均约 18 个原子，要回归 13 种量子化学性质。作者直接拿 Brockschmidt（2020）的实现，那份工作替多种 GNN 搜索了 500 组设置，大多数 GNN 的最佳层数是 8 层，暗示这个任务需要长距离的信息。\n\n作者用同样的程序、同样的切分和设置，只把最后一层改成 FA，每个性质训练 5 次。结果平均把错误率降低了 42%。\n\n他们接着排除了几个可能的解释：\n\n- **是不是 under-reaching？**至少 90% 的样本，层数已经大于等于分子的直径，远方节点本来就传得到。改用 10 层的模型，也没有比原本好。\n- **是不是向量不够大？**把维度加倍，只改善了 5.5%；FA 用原本的维度、不加权重，改善了 42%。\n- **一定要全连接吗？**最后一层只随机加上一半的边，错误率仍然平均降低 31.5%，加的边越多，改善越多。\n- **那干脆全部层都全连接？**错误率反而高了 1500%。图的结构还是需要的，只是最后要有一个让信息直接交流的地方。\n\n### 生物数据集\n\nNCI1 有 4,110 个化合物，要预测有没有抗肺癌活性；ENZYMES 有 600 个酶，要分成 6 类。作者沿用 Errica 等人（2020）的公平比较设置，10 折、3 个随机数种子，共 30 次的平均。\n\nErrica 等人当时发现，在 ENZYMES 上，完全不用图结构的基准比所有 GNN 都好。加上 FA 之后，GIN 在 ENZYMES 上比原本高 8.1 个百分点，也比不用图的基准高 2.5 个百分点；在 NCI1 上，GIN+FA 比原本最好的 GIN 高 1.5 个百分点。平均来说，FA 让错误率在 ENZYMES 降低 12%、在 NCI1 降低 4.8%。\n\n在 VarMisuse 程序数据集上，五种 GNN 加上 FA 之后，在训练时见过的项目上全都进步，R-GCN 达到 88.4%；在没见过的项目上，多数模型也进步了，R-GCN 从 82.9% 升到 83.8%，GGNN 和 GNN-MLP 则小幅下降。\n\n## 结论\n\n这篇论文把 GNN 处理长距离信息的困难，指向一个以前没被单独讨论的原因。感受野指数成长，向量长度固定，信息在瓶颈上被挤掉；GCN 和 GIN 这种一视同仁加总邻居的模型受影响最大。一个不加任何权重的全连接最后一层，就在大量调参过的模型上带来明显改善。\n\n作者明说 FA 只是起点。后来的研究沿着这个方向发展出不少做法，例如改变图的连接方式让瓶颈变宽，或是让每一层都同时做局部的消息传递与全局的注意力，GraphGPS 就是后者。\n\n\n> [!TIP]\n> \n> 把消息传递和全局注意力放在同一层的做法，可以参考我们的笔记：\n> \n> - **[[22.05] GraphGPS：图上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n如果你的 GNN 在长距离的任务上卡住了，加深、加宽之前，先在最后一层把所有节点连起来试试。不用加任何参数，改一下那一层的邻接矩阵就好。\n","imageSizes":{"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png":{"width":1078,"height":356},"/content/papers/assets/7a0066769f814f3963a4fbb407f58d7c16ecbc9fcc0c7107807c9ffd895b54e4.png":{"width":558,"height":316}}}