{"notes":[{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"多项式就好","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/chebnet.04caa9e465c7.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"空间上定义不出平移","text":"空间上定义不出平移","depth":3},{"id":"频域上算不起","text":"频域上算不起","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"用多项式换局部性","text":"用多项式换局部性","depth":3},{"id":"用-chebyshev-递推换速度","text":"用 Chebyshev 递推换速度","depth":3},{"id":"图上的池化","text":"图上的池化","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"先在网格上验证","text":"先在网格上验证","depth":3},{"id":"三种滤波器的比较","text":"三种滤波器的比较","depth":3},{"id":"图建得好不好","text":"图建得好不好","depth":3},{"id":"后来的事","text":"后来的事","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"绕道频域","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/spectral-networks.a3ae60467074.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"空间建构局部连接加聚类","text":"空间建构：局部连接加聚类","depth":3},{"id":"频域建构拉普拉斯的特征向量","text":"频域建构：拉普拉斯的特征向量","depth":3},{"id":"重新发现-cnn","text":"重新发现 CNN","depth":3},{"id":"平滑的系数局部的滤波器","text":"平滑的系数，局部的滤波器","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"次采样的-mnist","text":"次采样的 MNIST","depth":3},{"id":"球面上的-mnist","text":"球面上的 MNIST","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一个参数够吗？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/gcn.0866efaf54eb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"把图当成正则项","text":"把图当成正则项","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"从频域一路简化","text":"从频域一路简化","depth":3},{"id":"一层-gcn","text":"一层 GCN","depth":3},{"id":"两层就够","text":"两层就够","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"哪一种传播最好","text":"哪一种传播最好","depth":3},{"id":"叠深一点会怎样","text":"叠深一点会怎样","depth":3},{"id":"和-wl-算法的关系","text":"和 WL 算法的关系","depth":3},{"id":"四天后的一篇评论","text":"四天后的一篇评论","depth":3},{"id":"论文自己列的限制","text":"论文自己列的限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"CNN 在图像上好用，靠的是三件事。卷积核只看一小块区域，同一组权重在整张图上重复使用，一层一层叠上去还能组出越来越大的图样。\n\n这三件事都建立在网格上。像素有上下左右，「往右平移一格」有明确的意思。社交网络、脑部的连接图、词向量之间的相似度，数据都长在图上，每个节点的邻居数量不同，也没有顺序，平移这件事根本定义不出来。\n\nEPFL 的 Michaël Defferrard、Xavier Bresson 与 Pierre Vandergheynst 来自图信号处理的领域。Vandergheynst 早年和 Hammond 等人用 Chebyshev 多项式在图上做小波，这篇把同一套工具带进了 CNN。\n\n## 定义问题\n\n### 空间上定义不出平移\n\n要在图上定义卷积，第一个想法是直接在空间上做，像图像那样，拿一个固定大小的核去对齐每个节点的邻居。作者指出，这条路得先解决「邻居怎么对齐」的问题；在图上，从空间的角度，没有唯一的数学定义能描述平移。\n\n另一条路是绕到频域。卷积定理说，卷积是在傅里叶基底下会被对角化的线性运算。图上也有傅里叶基底，就是拉普拉斯矩阵的特征向量。\n\n### 频域上算不起\n\n组合拉普拉斯 $L=D-W$ 或归一化拉普拉斯 $L=I_n-D^{-1/2}WD^{-1/2}$ 都是实对称半正定矩阵，可以拆成 $L=U\\Lambda U^\\top$。$U$ 的每一列是一个「傅里叶模态」，特征值 $\\lambda_l$ 就是它的频率。信号 $x\\in\\mathbb{R}^n$ 的图傅里叶变换是 $\\hat x=U^\\top x$。\n\n于是滤波可以写成：\n\n$$\ny=g_\\theta(L)x=U\\,g_\\theta(\\Lambda)\\,U^\\top x\n$$\n\n最直接的做法是让每个频率的系数都自由学，$g_\\theta(\\Lambda)=\\mathrm{diag}(\\theta)$。这样有两个问题。滤波器在空间上不局部，可能一下子牵动整张图；参数数量是 $O(n)$，跟节点数一样多。\n\n就算把参数数量压下来，计算还是很贵。2014 年 Bruna 等人的频域网络用平滑的样条函数参数化频率系数，换取空间上的局部性。但每次前向、反向传递都要乘两次 $U$，各是 $O(n^2)$；$U$ 本身要先对 $L$ 做特征分解，还得存下一个 $n\\times n$ 的稠密矩阵。\n\n\n> [!TIP]\n> \n> Bruna 等人的空间与频域两种建构，可以参考我们的笔记：\n> \n> - **[[13.12] Spectral Networks：绕道频域](/blog/papers/graph-learning/spectral-networks)**\n\n\n这篇要解决的，是一个在图上**严格局部**、而且**算得快**的滤波器。\n\n## 解决问题\n\n### 用多项式换局部性\n\n作者把滤波器限制成 $\\Lambda$ 的多项式：\n\n$$\ng_\\theta(\\Lambda)=\\sum_{k=0}^{K-1}\\theta_k\\Lambda^k\n$$\n\n因为 $(U\\Lambda U^\\top)^k=U\\Lambda^kU^\\top$，这等于直接对拉普拉斯矩阵取多项式 $\\sum_k\\theta_kL^k$。$L$ 只在相邻节点之间有非零值，$L^k$ 就只连得到 $k$ 步以内的节点；两个节点的最短距离大于 $k$，$(L^k)_{ij}$ 就是 0。\n\n拿一条七个节点的路径试试看，在正中间放一个 1，看 $L^k$ 会碰到哪些节点：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\n# 一条 7 个节点的路径：0-1-2-3-4-5-6\nn = 7\nA = np.zeros((n, n))\nfor i in range(n - 1):\n    A[i, i + 1] = A[i + 1, i] = 1\nd = A.sum(1)\nL = np.eye(n) - A / np.sqrt(np.outer(d, d))   # 归一化拉普拉斯\n\ndelta = np.zeros(n)\ndelta[3] = 1                                   # 只在正中间的节点放一个 1\nfor k in range(4):\n    print(f\"L^{k} δ:\", (np.linalg.matrix_power(L, k) @ delta != 0).astype(int))\n```\n\n```text\nL^0 δ: [0 0 0 1 0 0 0]\nL^1 δ: [0 0 1 1 1 0 0]\nL^2 δ: [0 1 1 1 1 1 0]\nL^3 δ: [1 1 1 1 1 1 1]\n```\n\n每多一次方，影响范围就往外扩一步。$K-1$ 阶的多项式，影响范围刚好是 $K-1$ 步以内，参数也只有 $K$ 个，和节点数无关。这和图像上 5×5 的卷积核只看 5×5 的范围是同一件事。\n\n### 用 Chebyshev 递推换速度\n\n局部性有了，但如果还是先特征分解、再在频域里乘，成本依旧是 $O(n^2)$。\n\n作者改用 Chebyshev 多项式 $T_k$ 当基底。它满足一个稳定的递推关系：\n\n$$\nT_k(x)=2xT_{k-1}(x)-T_{k-2}(x),\\qquad T_0=1,\\quad T_1=x\n$$\n\nChebyshev 多项式定义在 $[-1,1]$，所以先把拉普拉斯缩放成 $\\tilde L=2L/\\lambda_{\\max}-I_n$，滤波器写成：\n\n$$\ny=\\sum_{k=0}^{K-1}\\theta_kT_k(\\tilde L)\\,x\n$$\n\n令 $\\bar x_k=T_k(\\tilde L)x$，就有 $\\bar x_k=2\\tilde L\\bar x_{k-1}-\\bar x_{k-2}$。整个滤波只需要 $K$ 次稀疏矩阵乘向量，成本是 $O(K|E|)$。不用特征分解，也不用存 $U$。\n\n接着上面的路径，用两种方法算同一个 $K=3$ 的滤波器，确认答案一样：\n\n```python\ntheta = np.array([0.5, -1.0, 0.3])            # K = 3，三个 Chebyshev 系数\nx = np.random.default_rng(0).random(n)\n\n# 做法一：特征分解，在频域里乘上滤波器\nlam, U = np.linalg.eigh(L)\nlam_max = lam.max()\nlam_s = 2 * lam / lam_max - 1                  # 把频率缩放到 [-1, 1]\ng = theta[0] + theta[1] * lam_s + theta[2] * (2 * lam_s**2 - 1)   # T0, T1, T2\ny_spec = U @ (g * (U.T @ x))\n\n# 做法二：Chebyshev 递推，只做稀疏矩阵乘法\nL_s = 2 * L / lam_max - np.eye(n)\nx0, x1 = x, L_s @ x\nx2 = 2 * L_s @ x1 - x0\ny_rec = theta[0] * x0 + theta[1] * x1 + theta[2] * x2\n\nprint(\"频域:\", y_spec)\nprint(\"递推:\", y_rec)\n```\n\n```text\n频域: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\n递推: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\n```\n\n示范里为了方便比对，$\\lambda_{\\max}$ 是用特征分解求的；实际上只需要估最大特征值，归一化拉普拉斯的特征值也一定不超过 2。\n\n多个输入、输出通道时，第 $j$ 个输出特征图是 $y_j=\\sum_i g_{\\theta_{ij}}(L)x_i$，每一对输入输出通道各有一组 $K$ 个系数。反向传播同样只需要 $K$ 次稀疏矩阵乘法。\n\n### 图上的池化\n\nCNN 还需要池化。作者用 Graclus 多层聚类算法做粗化，每一层把相邻的节点两两合并，节点数大约减半；配对时挑让 normalized cut 最大的邻居。\n\n粗化之后，节点的顺序没有任何意义，池化时得查表才知道谁和谁合并，又慢又难并行化。作者的办法是补上一些不连任何边的「假节点」，让每个节点刚好有两个子节点，排成一棵平衡二元树，再依树的结构重新排列节点。排好之后，图上的池化就变成一维信号的池化，相邻的两个数字合并成一个。\n\n[![原论文 Figure 1：从输入的图信号开始，经过图卷积层（卷积、非线性）与图粗化（采样、池化）提取特征，再接全连接层做分类，输出标签。图中也画出频域滤波器在不同频率上的响应。](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)\n\n*原论文 Figure 1（Defferrard 等人），取自 [**PDF 第 2 页**](https://arxiv.org/pdf/1606.09375v3#page=2)。图注未收录。*\n\n整个网络就是图卷积、非线性、粗化、池化的组合，最后接全连接层，和 LeNet 的骨架一样。\n\n## 讨论\n\n### 先在网格上验证\n\n第一个实验是 MNIST。把 28×28 的像素网格建成 8 近邻图，加上假节点后共 976 个节点、3,198 条边。网络照 TensorFlow 教学里的 LeNet-5 架构，图像版用 5×5 卷积，图版用 $K=25$ 的滤波器，两者看的范围相当。\n\n| 模型 | 架构 | 准确率 |\n| --- | --- | ---: |\n| 一般 CNN | C32-P4-C64-P4-FC512 | 99.33% |\n| 图 CNN | GC32-P4-GC64-P4-FC512 | 99.14% |\n\n差距很小。作者自己解释，可能来自频域滤波器的**各向同性**，图上的边没有上下左右的方向；这到底是缺点还是优点，要看任务，就像旋转不变性有时候正是我们想要的。\n\n### 三种滤波器的比较\n\n同样在 MNIST 上，比较三种频域滤波器（$K=25$）：\n\n| 架构 | 自由系数 | 样条（Bruna 等人） | Chebyshev |\n| --- | ---: | ---: | ---: |\n| GC10 | 95.75% | 97.26% | 97.48% |\n| GC32-P4-GC64-P4-FC512 | 96.28% | 97.15% | 99.14% |\n\n自由系数的滤波器不局部，参数又多，表现最差。Chebyshev 在两种架构上都最好。\n\n速度方面，Bruna 等人的方法随节点数以 $O(n^2)$ 成长，这篇是 $O(n)$。在 20NEWS 上把词数从 2,000 增加到 12,000，旧方法处理一个批量的时间从几十毫秒爬到 1.2 秒左右，Chebyshev 只增加到 150 毫秒上下。\n\n不过和一般 CNN 比，图 CNN 还是慢了不少。一批 100 张 MNIST，CNN 在 GPU 上 31 毫秒，图 CNN 要 200 毫秒；搬到 GPU 上的加速比倒是差不多，分别是 6.77 倍与 8 倍。\n\n### 图建得好不好\n\n20NEWS 是文本分类，节点是 10,000 个常用词，用 word2vec 向量建 16 近邻图。图 CNN 的准确率是 68.26%，赢过全连接网络，但没有赢过多项式 Naive Bayes 的 68.51%。\n\n作者花了一节谈图的质量。把 MNIST 的网格图换成随机图，同一个五层架构从 99.14% 掉到 95.39%。卷积的前提是局部性、平稳性和组合性，数据所在的图得满足这些假设，卷积才有意义。\n\n### 后来的事\n\n三个月后，Kipf 和 Welling 的 GCN 以这篇为基础，把多项式砍到一阶，再把两个系数绑成一个。在引用网络的半监督分类上，GCN 比 $K=2$、$K=3$ 的 Chebyshev 滤波器还准，Pubmed 上差了 5 个百分点左右。\n\nGCN 发表四天后，Ferenc Huszár 在博客评论 GCN 时说，简化太过头会让模型在图像网格上只剩一个固定的核；相较之下，ChebNet 的高阶多项式虽然计算成本较高，看起来更有前景。[inFERENCe，2016-09-13](https://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/)\n\n\n> [!TIP]\n> \n> GCN 的简化推导与那场讨论，可以参考我们的笔记：\n> \n> - **[[16.09] GCN：一个参数够吗？](/blog/papers/graph-learning/gcn)**\n\n\n## 结论\n\nChebNet 把频域图卷积从理论变成可以训练的网络。多项式让滤波器只看 $K-1$ 步以内的邻居，Chebyshev 递推让计算只需要稀疏矩阵乘法，不必做特征分解。在 MNIST 上，它和同架构的 CNN 只差 0.19 个百分点。\n\n$K$ 是这个方法最重要的超参数。它同时决定每层看多远、要学几个系数；在引用网络这类图上，GCN 的实验显示 $K$ 小一点反而比较准，用之前值得先扫几个值。\n\n频域这条路后来大多走向 GCN 那样的一阶近似。不过拉普拉斯的特征向量并没有退场，几年后的 Graph Transformer，又把它们捡回来当成节点的位置编码。\n","imageSizes":{"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png":{"width":1144,"height":484}}}