{"notes":[{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"绕道频域","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/spectral-networks.a3ae60467074.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"空间建构局部连接加聚类","text":"空间建构：局部连接加聚类","depth":3},{"id":"频域建构拉普拉斯的特征向量","text":"频域建构：拉普拉斯的特征向量","depth":3},{"id":"重新发现-cnn","text":"重新发现 CNN","depth":3},{"id":"平滑的系数局部的滤波器","text":"平滑的系数，局部的滤波器","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"次采样的-mnist","text":"次采样的 MNIST","depth":3},{"id":"球面上的-mnist","text":"球面上的 MNIST","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"多项式就好","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/chebnet.04caa9e465c7.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"空间上定义不出平移","text":"空间上定义不出平移","depth":3},{"id":"频域上算不起","text":"频域上算不起","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"用多项式换局部性","text":"用多项式换局部性","depth":3},{"id":"用-chebyshev-递推换速度","text":"用 Chebyshev 递推换速度","depth":3},{"id":"图上的池化","text":"图上的池化","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"先在网格上验证","text":"先在网格上验证","depth":3},{"id":"三种滤波器的比较","text":"三种滤波器的比较","depth":3},{"id":"图建得好不好","text":"图建得好不好","depth":3},{"id":"后来的事","text":"后来的事","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一个参数够吗？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/gcn.0866efaf54eb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"把图当成正则项","text":"把图当成正则项","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"从频域一路简化","text":"从频域一路简化","depth":3},{"id":"一层-gcn","text":"一层 GCN","depth":3},{"id":"两层就够","text":"两层就够","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"哪一种传播最好","text":"哪一种传播最好","depth":3},{"id":"叠深一点会怎样","text":"叠深一点会怎样","depth":3},{"id":"和-wl-算法的关系","text":"和 WL 算法的关系","depth":3},{"id":"四天后的一篇评论","text":"四天后的一篇评论","depth":3},{"id":"论文自己列的限制","text":"论文自己列的限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"2013 年底，AlexNet 刚让 CNN 在图像识别上大获全胜一年多。纽约大学的 Joan Bruna、Wojciech Zaremba、Yann LeCun 与纽约市立学院的 Arthur Szlam 想知道，卷积能不能用在不是网格的数据上。\n\n气象站的观测网、3D 网格上的温度、社交网络、推荐系统，这些数据都有某种「谁和谁相近」的结构，但没有规则的上下左右。\n\n这篇论文提出了两种在图上建构 CNN 的方法。三年后的 ChebNet 和 GCN，都是从其中的频域建构往下简化；GCN 论文也写明，自己的方法建立在这篇提出的频域图卷积上。\n\n## 定义问题\n\n作者先拆解 CNN 为什么省参数。在网格上，CNN 同时用了三种结构：\n\n1. **平移**：同一个滤波器在每个位置重复使用，也就是权重共享。\n2. **距离**：滤波器只看附近一小块，支撑范围远小于整张输入。\n3. **多尺度的二分聚类**：网格可以一层层对半合并，才有步长卷积与池化。\n\n如果输入有 $n$ 个坐标，一个输出 $m$ 维的全连接层需要 $n\\cdot m$ 个参数，通常是 $O(n^2)$。只用平移或只用局部连接，可以降到每张特征图 $O(n)$；两者一起用，参数变成 $O(k\\cdot S)$，$k$ 是特征图数、$S$ 是滤波器大小，和 $n$ 无关。\n\n图上没有平移，也不一定有好的坐标。作者想知道，只剩下一张带权重的图 $W$，这三种结构能捡回多少。\n\n## 解决问题\n\n### 空间建构：局部连接加聚类\n\n第一种方法直接在空间上做。图的权重本身就定义了「附近」，例如设一个门槛 $\\delta$，把 $W_{ij}>\\delta$ 的节点当成邻居。滤波器只连到邻居，参数就从 $O(n^2)$ 降到 $O(S\\cdot n)$，$S$ 是平均邻居数。\n\n接着在图上做多尺度的层次聚类，第 $k$ 层把节点分成 $d_k$ 群。每一层先做只连邻居的稀疏滤波，再在每一群里做池化：\n\n$$\nx_{k+1,j}=L_k\\,h\\left(\\sum_{i=1}^{f_{k-1}}F_{k,i,j}\\,x_{k,i}\\right)\n$$\n\n$F_{k,i,j}$ 是一个只在邻居位置有值的稀疏矩阵，$h$ 是非线性，$L_k$ 是对每一群做池化。\n\n[![原论文 Figure 2：三个分辨率的空间建构。最左边的灰色节点是原始的图，红色区块代表第一层的局部滤波与聚类，得到中间较少的红色节点；绿色区块再做一次，得到最右边的绿色节点，每一层节点变少、特征维度变多。](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)\n\n*原论文 Figure 2（Bruna 等人），取自 [**PDF 第 4 页**](https://arxiv.org/pdf/1312.6203v3#page=4)。图注未收录。*\n\n每往右一层，空间分辨率下降，特征维度增加，和 CNN 的金字塔一样。\n\n这个方法对图的要求很低，但有一个根本的限制。不同位置的滤波器各学各的，没有办法共享权重。参数量是 $O(n)$，不是 $O(1)$。\n\n### 频域建构：拉普拉斯的特征向量\n\n第二种方法绕道频域。作者用组合拉普拉斯 $L=D-W$ 来衡量信号在图上的平滑程度：\n\n$$\n\\left\\|\\nabla x\\right\\|_W^2=\\sum_i\\sum_jW_{ij}\\left[x(i)-x(j)\\right]^2\n$$\n\n最平滑的向量是常数向量。在与它正交的向量里找最平滑的，再找与前两者正交的最平滑向量，一路找下去，得到的就是 $L$ 的特征向量 $v_0, v_1, \\dots$。特征值越小，向量越平滑。这和网格上的傅里叶基底是同一回事，网格上拉普拉斯的特征向量就是正弦波。\n\n于是卷积可以定义成在这组基底上做对角运算：\n\n$$\nx_{k+1,j}=h\\left(V\\sum_{i=1}^{f_{k-1}}F_{k,i,j}V^\\top x_{k,i}\\right)\n$$\n\n$V$ 是特征向量矩阵，$F_{k,i,j}$ 是对角矩阵，每个频率乘一个系数。实际计算时只保留前 $d$ 个低频的特征向量，它们带着图平滑的几何结构。这样每一层的参数是 $f_{k-1}\\cdot f_k\\cdot d$，仍然是 $O(n)$。\n\n作者也坦白列了几个麻烦。多数图只有最低频的那一小段特征向量有意义；前向和反向传递都要乘上稠密的 $V$ 与 $V^\\top$，很贵；非线性得在空间那一侧做，不能留在频域里。\n\n### 重新发现 CNN\n\n论文有一段很漂亮的观察。如果拿数据的协方差矩阵当图的权重，对自然图像来说，平移不变性让协方差矩阵只和两个像素的位移有关，它的特征向量就是傅里叶基底，大致依频率排好。这时，频域建构里的 $VF V^\\top$ 都在傅里叶基底下是对角的，也就是平移不变的，正好是一般的卷积。\n\n换句话说，不必事先知道图像是网格，只要从数据里建图，这套方法就能把标准 CNN 找回来。\n\n### 平滑的系数，局部的滤波器\n\n要把参数降到 $O(1)$，作者借用网格上的对偶关系。信号在空间上衰减得快，在频域里就平滑。反过来，如果滤波器的频率响应是平滑的，它在空间上就会比较集中。\n\n所以作者不让每个频率的系数自由学，而是把特征向量依特征值排成一维，只学少数几个控制点，中间用三次样条插值：\n\n$$\n\\mathrm{diag}\\left(F_{k,i,j}\\right)=\\mathcal K\\,\\alpha_{k,i,j}\n$$\n\n$\\mathcal K$ 是固定的三次样条核，$\\alpha$ 是要学的系数。控制点的数量不必随 $n$ 增加，参数就是 $O(1)$。\n\n拿一条 64 个节点的路径比一比。在正中间放一个脉冲，分别用 64 个自由系数和 6 个控制点插值出的平滑系数去滤波，看响应散到多少个节点：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(0)\n\nn = 64                                     # 一条 64 个节点的路径\nW = np.zeros((n, n))\nfor i in range(n - 1):\n    W[i, i + 1] = W[i + 1, i] = 1\nL = np.diag(W.sum(1)) - W                  # 组合拉普拉斯 L = D - W\nlam, V = np.linalg.eigh(L)                 # 由低频排到高频\n\nx = np.zeros(n)\nx[n // 2] = 1                              # 在正中间放一个脉冲\n\ndef spread(y, ratio=0.05):\n    return int((np.abs(y) > ratio * np.abs(y).max()).sum())   # 有多少节点的响应超过最大值的 5%\n\nfrom scipy.interpolate import CubicSpline\n\nfree = rng.normal(size=n)                                   # 每个频率一个自由系数，共 64 个\nknots = np.linspace(0, n - 1, 6)\nsmooth = CubicSpline(knots, rng.normal(size=6))(np.arange(n))  # 只学 6 个控制点，其余用三次样条插值\n\nfor name, g, params in [(\"自由系数\", free, n), (\"平滑系数\", smooth, 6)]:\n    y = V @ (g * (V.T @ x))                                 # 滤波：V diag(g) V^T x\n    print(f\"{name}：{params:>2} 个参数，响应扩散到 {spread(y):>2} 个节点\")\n```\n\n```text\n自由系数：64 个参数，响应扩散到 60 个节点\n平滑系数： 6 个参数，响应扩散到 15 个节点\n```\n\n自由系数的滤波器几乎碰到整条路径；平滑系数的滤波器集中在脉冲附近。换几个随机数种子，趋势都一样。\n\n不过平滑只是让滤波器「比较」局部，并没有保证它只碰到几步以内的节点。这个缺口，后来由 ChebNet 用多项式补上。\n\n## 讨论\n\n### 次采样的 MNIST\n\n第一个实验把 28×28 的 MNIST 随机采样成 400 个坐标。这些点还是在二维平面上，但已经不是网格，一般的卷积用不了。\n\n| 方法 | 参数量 | 错误率 |\n| --- | ---: | ---: |\n| 最近邻 | — | 4.11% |\n| 全连接 400-FC800-FC50-10 | $3.6\\times10^5$ | 1.8% |\n| 空间建构，一层 | $7.2\\times10^4$ | 1.8% |\n| 空间建构，两层 | $1.6\\times10^5$ | **1.3%** |\n| 频域建构，自由系数 | $3.2\\times10^3$ | 2.6% |\n| 频域建构，平滑系数（$q=32$） | $1.6\\times10^3$ | 2.3% |\n| 频域建构，平滑系数（$q=20$，较宽） | $5\\times10^3$ | 1.8% |\n\n空间建构的两层网络最好，错误率 1.3%。频域建构差一些，但参数只有全连接的百分之一左右；加上平滑之后，错误率从 2.6% 降到 2.3%，加宽后到 1.8%。\n\n作者的解释是，MNIST 的数字是由局部、有方向的笔画构成的，需要空间上很集中的测量。局部连接本来就是局部的；频域建构不保证局部，加上平滑之后才变得比较集中。\n\n[![原论文 Figure 6：六张小图，每张是 400 个采样点上一个滤波器的响应，颜色代表数值。(a)(b) 是空间建构在两个不同群里学到的滤波器，只在一小块区域有颜色；(c)(d) 是自由系数的频域滤波器，颜色散布在整张图；(e)(f) 是平滑系数的频域滤波器，颜色集中在局部。](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)\n\n*原论文 Figure 6（Bruna 等人），取自 [**PDF 第 10 页**](https://arxiv.org/pdf/1312.6203v3#page=10)。图注未收录。*\n\n这张图把三种滤波器并排。(a)(b) 是空间建构，两个不同的群各自学了一个看起来相同的特征，因为它们没有共享权重。(c)(d) 是自由系数的频域滤波器，响应散布在整张图上，作者说这正是傅里叶分析在描述局部现象时的根本限制。(e)(f) 加了平滑之后，响应重新集中，而且同一个滤波器可以在不同位置检测类似的特征，这才像卷积。\n\n### 球面上的 MNIST\n\n第二个实验把 MNIST 投影到单位球面上的 4,096 个随机点，再对每张图做随机的旋转。\n\n轻微旋转时，最近邻的错误率升到 19%，各种神经网络都在 5.6% 到 7% 之间。卷积式的建构和全连接打平，参数却少得多。全连接约 $10^7$ 个参数、5.6%；平滑系数的频域建构约 $9\\times10^5$ 个参数、6%。\n\n旋转完全随机时，最近邻错了 80%，全连接 52%，平滑频域建构 50%，是表中最好的，但所有方法都不理想。作者认为这需要完整的旋转加平移不变性，而三维旋转群不可交换，可能要更深的架构。\n\n## 结论\n\n这篇论文提出了两条路。空间建构直接在图上做局部连接和聚类池化，参数降到 $O(n)$，但没有权重共享；频域建构用拉普拉斯的特征向量当傅里叶基底，再用平滑的频率系数换取空间上的局部性，参数可以降到 $O(1)$。\n\n频域这条路的代价也一起留了下来。它要做特征分解，每次转换都乘一个 $n\\times n$ 的稠密矩阵，局部性也只是近似。三年后，ChebNet 把平滑的样条换成拉普拉斯的多项式，保证局部，也不必特征分解；再三个月，GCN 把多项式砍到一阶。\n\n\n> [!TIP]\n> \n> 接下来两步的简化，可以参考我们的笔记：\n> \n> - **[[16.06] ChebNet：多项式就好](/blog/papers/graph-learning/chebnet)**\n> - **[[16.09] GCN：一个参数够吗？](/blog/papers/graph-learning/gcn)**\n\n\n论文里那个从协方差矩阵找回 CNN 的例子，到现在读起来还是很有意思。卷积可以看成在某组基底下的对角运算，网格只是刚好提供了那组基底；在图上，这组基底得自己去找。\n","imageSizes":{"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png":{"width":1210,"height":448},"/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png":{"width":908,"height":1208}}}