{"notes":[{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"繞道頻域","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/spectral-networks.31f7482724ab.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"空間建構局部連接加分群","text":"空間建構：局部連接加分群","depth":3},{"id":"頻域建構拉普拉斯的特徵向量","text":"頻域建構：拉普拉斯的特徵向量","depth":3},{"id":"重新發現-cnn","text":"重新發現 CNN","depth":3},{"id":"平滑的係數局部的濾波器","text":"平滑的係數，局部的濾波器","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"次取樣的-mnist","text":"次取樣的 MNIST","depth":3},{"id":"球面上的-mnist","text":"球面上的 MNIST","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"多項式就好","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/chebnet.6954c188a205.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"空間上定義不出平移","text":"空間上定義不出平移","depth":3},{"id":"頻域上算不起","text":"頻域上算不起","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"用多項式換局部性","text":"用多項式換局部性","depth":3},{"id":"用-chebyshev-遞迴換速度","text":"用 Chebyshev 遞迴換速度","depth":3},{"id":"圖上的池化","text":"圖上的池化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"先在網格上驗證","text":"先在網格上驗證","depth":3},{"id":"三種濾波器的比較","text":"三種濾波器的比較","depth":3},{"id":"圖建得好不好","text":"圖建得好不好","depth":3},{"id":"後來的事","text":"後來的事","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一個參數夠嗎？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/gcn.66a3d9542810.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"把圖當成正則項","text":"把圖當成正則項","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"從頻域一路簡化","text":"從頻域一路簡化","depth":3},{"id":"一層-gcn","text":"一層 GCN","depth":3},{"id":"兩層就夠","text":"兩層就夠","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"哪一種傳播最好","text":"哪一種傳播最好","depth":3},{"id":"疊深一點會怎樣","text":"疊深一點會怎樣","depth":3},{"id":"和-wl-演算法的關係","text":"和 WL 演算法的關係","depth":3},{"id":"四天後的一篇評論","text":"四天後的一篇評論","depth":3},{"id":"論文自己列的限制","text":"論文自己列的限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"2013 年底，AlexNet 剛讓 CNN 在影像辨識上大獲全勝一年多。紐約大學的 Joan Bruna、Wojciech Zaremba、Yann LeCun 與紐約市立學院的 Arthur Szlam 想知道，卷積能不能用在不是網格的資料上。\n\n氣象站的觀測網、3D 網格上的溫度、社群網路、推薦系統，這些資料都有某種「誰和誰相近」的結構，但沒有規則的上下左右。\n\n這篇論文提出了兩種在圖上建構 CNN 的方法。三年後的 ChebNet 和 GCN，都是從其中的頻域建構往下簡化；GCN 論文也寫明，自己的方法建立在這篇提出的頻域圖卷積上。\n\n## 定義問題\n\n作者先拆解 CNN 為什麼省參數。在網格上，CNN 同時用了三種結構：\n\n1. **平移**：同一個濾波器在每個位置重複使用，也就是權重共享。\n2. **距離**：濾波器只看附近一小塊，支撐範圍遠小於整張輸入。\n3. **多尺度的二分分群**：網格可以一層層對半合併，才有步長卷積與池化。\n\n如果輸入有 $n$ 個座標，一個輸出 $m$ 維的全連接層需要 $n\\cdot m$ 個參數，通常是 $O(n^2)$。只用平移或只用局部連接，可以降到每張特徵圖 $O(n)$；兩者一起用，參數變成 $O(k\\cdot S)$，$k$ 是特徵圖數、$S$ 是濾波器大小，和 $n$ 無關。\n\n圖上沒有平移，也不一定有好的座標。作者想知道，只剩下一張帶權重的圖 $W$，這三種結構能撿回多少。\n\n## 解決問題\n\n### 空間建構：局部連接加分群\n\n第一種方法直接在空間上做。圖的權重本身就定義了「附近」，例如設一個門檻 $\\delta$，把 $W_{ij}>\\delta$ 的節點當成鄰居。濾波器只連到鄰居，參數就從 $O(n^2)$ 降到 $O(S\\cdot n)$，$S$ 是平均鄰居數。\n\n接著在圖上做多尺度的階層分群，第 $k$ 層把節點分成 $d_k$ 群。每一層先做只連鄰居的稀疏濾波，再在每一群裡做池化：\n\n$$\nx_{k+1,j}=L_k\\,h\\left(\\sum_{i=1}^{f_{k-1}}F_{k,i,j}\\,x_{k,i}\\right)\n$$\n\n$F_{k,i,j}$ 是一個只在鄰居位置有值的稀疏矩陣，$h$ 是非線性，$L_k$ 是對每一群做池化。\n\n[![原論文 Figure 2：三個解析度的空間建構。最左邊的灰色節點是原始的圖，紅色區塊代表第一層的局部濾波與分群，得到中間較少的紅色節點；綠色區塊再做一次，得到最右邊的綠色節點，每一層節點變少、特徵維度變多。](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)\n\n*原論文 Figure 2（Bruna 等人），取自 [**PDF 第 4 頁**](https://arxiv.org/pdf/1312.6203v3#page=4)。圖說未收錄。*\n\n每往右一層，空間解析度下降，特徵維度增加，和 CNN 的金字塔一樣。\n\n這個方法對圖的要求很低，但有一個根本的限制。不同位置的濾波器各學各的，沒有辦法共享權重。參數量是 $O(n)$，不是 $O(1)$。\n\n### 頻域建構：拉普拉斯的特徵向量\n\n第二種方法繞道頻域。作者用組合拉普拉斯 $L=D-W$ 來衡量訊號在圖上的平滑程度：\n\n$$\n\\left\\|\\nabla x\\right\\|_W^2=\\sum_i\\sum_jW_{ij}\\left[x(i)-x(j)\\right]^2\n$$\n\n最平滑的向量是常數向量。在與它正交的向量裡找最平滑的，再找與前兩者正交的最平滑向量，一路找下去，得到的就是 $L$ 的特徵向量 $v_0, v_1, \\dots$。特徵值越小，向量越平滑。這和網格上的傅立葉基底是同一回事，網格上拉普拉斯的特徵向量就是正弦波。\n\n於是卷積可以定義成在這組基底上做對角運算：\n\n$$\nx_{k+1,j}=h\\left(V\\sum_{i=1}^{f_{k-1}}F_{k,i,j}V^\\top x_{k,i}\\right)\n$$\n\n$V$ 是特徵向量矩陣，$F_{k,i,j}$ 是對角矩陣，每個頻率乘一個係數。實務上只保留前 $d$ 個低頻的特徵向量，它們帶著圖平滑的幾何結構。這樣每一層的參數是 $f_{k-1}\\cdot f_k\\cdot d$，仍然是 $O(n)$。\n\n作者也坦白列了幾個麻煩。多數圖只有最低頻的那一小段特徵向量有意義；前向和反向傳遞都要乘上稠密的 $V$ 與 $V^\\top$，很貴；非線性得在空間那一側做，不能留在頻域裡。\n\n### 重新發現 CNN\n\n論文有一段很漂亮的觀察。如果拿資料的共變異數矩陣當圖的權重，對自然影像來說，平移不變性讓共變異數矩陣只和兩個像素的位移有關，它的特徵向量就是傅立葉基底，大致依頻率排好。這時，頻域建構裡的 $VF V^\\top$ 都在傅立葉基底下是對角的，也就是平移不變的，正好是一般的卷積。\n\n換句話說，不必事先知道影像是網格，只要從資料裡建圖，這套方法就能把標準 CNN 找回來。\n\n### 平滑的係數，局部的濾波器\n\n要把參數降到 $O(1)$，作者借用網格上的對偶關係。訊號在空間上衰減得快，在頻域裡就平滑。反過來，如果濾波器的頻率響應是平滑的，它在空間上就會比較集中。\n\n所以作者不讓每個頻率的係數自由學，而是把特徵向量依特徵值排成一維，只學少數幾個控制點，中間用三次樣條內插：\n\n$$\n\\mathrm{diag}\\left(F_{k,i,j}\\right)=\\mathcal K\\,\\alpha_{k,i,j}\n$$\n\n$\\mathcal K$ 是固定的三次樣條核，$\\alpha$ 是要學的係數。控制點的數量不必隨 $n$ 增加，參數就是 $O(1)$。\n\n拿一條 64 個節點的路徑比一比。在正中間放一個脈衝，分別用 64 個自由係數和 6 個控制點內插出的平滑係數去濾波，看響應散到多少個節點：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(0)\n\nn = 64                                     # 一條 64 個節點的路徑\nW = np.zeros((n, n))\nfor i in range(n - 1):\n    W[i, i + 1] = W[i + 1, i] = 1\nL = np.diag(W.sum(1)) - W                  # 組合拉普拉斯 L = D - W\nlam, V = np.linalg.eigh(L)                 # 由低頻排到高頻\n\nx = np.zeros(n)\nx[n // 2] = 1                              # 在正中間放一個脈衝\n\ndef spread(y, ratio=0.05):\n    return int((np.abs(y) > ratio * np.abs(y).max()).sum())   # 有多少節點的響應超過最大值的 5%\n\nfrom scipy.interpolate import CubicSpline\n\nfree = rng.normal(size=n)                                   # 每個頻率一個自由係數，共 64 個\nknots = np.linspace(0, n - 1, 6)\nsmooth = CubicSpline(knots, rng.normal(size=6))(np.arange(n))  # 只學 6 個控制點，其餘用三次樣條內插\n\nfor name, g, params in [(\"自由係數\", free, n), (\"平滑係數\", smooth, 6)]:\n    y = V @ (g * (V.T @ x))                                 # 濾波：V diag(g) V^T x\n    print(f\"{name}：{params:>2} 個參數，響應擴散到 {spread(y):>2} 個節點\")\n```\n\n```text\n自由係數：64 個參數，響應擴散到 60 個節點\n平滑係數： 6 個參數，響應擴散到 15 個節點\n```\n\n自由係數的濾波器幾乎碰到整條路徑；平滑係數的濾波器集中在脈衝附近。換幾個亂數種子，趨勢都一樣。\n\n不過平滑只是讓濾波器「比較」局部，並沒有保證它只碰到幾步以內的節點。這個缺口，後來由 ChebNet 用多項式補上。\n\n## 討論\n\n### 次取樣的 MNIST\n\n第一個實驗把 28×28 的 MNIST 隨機取樣成 400 個座標。這些點還是在二維平面上，但已經不是網格，一般的卷積用不了。\n\n| 方法 | 參數量 | 錯誤率 |\n| --- | ---: | ---: |\n| 最近鄰 | — | 4.11% |\n| 全連接 400-FC800-FC50-10 | $3.6\\times10^5$ | 1.8% |\n| 空間建構，一層 | $7.2\\times10^4$ | 1.8% |\n| 空間建構，兩層 | $1.6\\times10^5$ | **1.3%** |\n| 頻域建構，自由係數 | $3.2\\times10^3$ | 2.6% |\n| 頻域建構，平滑係數（$q=32$） | $1.6\\times10^3$ | 2.3% |\n| 頻域建構，平滑係數（$q=20$，較寬） | $5\\times10^3$ | 1.8% |\n\n空間建構的兩層網路最好，錯誤率 1.3%。頻域建構差一些，但參數只有全連接的百分之一左右；加上平滑之後，錯誤率從 2.6% 降到 2.3%，加寬後到 1.8%。\n\n作者的解釋是，MNIST 的數字是由局部、有方向的筆畫構成的，需要空間上很集中的量測。局部連接本來就是局部的；頻域建構不保證局部，加上平滑之後才變得比較集中。\n\n[![原論文 Figure 6：六張小圖，每張是 400 個取樣點上一個濾波器的響應，顏色代表數值。(a)(b) 是空間建構在兩個不同群裡學到的濾波器，只在一小塊區域有顏色；(c)(d) 是自由係數的頻域濾波器，顏色散布在整張圖；(e)(f) 是平滑係數的頻域濾波器，顏色集中在局部。](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)\n\n*原論文 Figure 6（Bruna 等人），取自 [**PDF 第 10 頁**](https://arxiv.org/pdf/1312.6203v3#page=10)。圖說未收錄。*\n\n這張圖把三種濾波器並排。(a)(b) 是空間建構，兩個不同的群各自學了一個看起來相同的特徵，因為它們沒有共享權重。(c)(d) 是自由係數的頻域濾波器，響應散布在整張圖上，作者說這正是傅立葉分析在描述局部現象時的根本限制。(e)(f) 加了平滑之後，響應重新集中，而且同一個濾波器可以在不同位置偵測類似的特徵，這才像卷積。\n\n### 球面上的 MNIST\n\n第二個實驗把 MNIST 投影到單位球面上的 4,096 個隨機點，再對每張圖做隨機的旋轉。\n\n輕微旋轉時，最近鄰的錯誤率升到 19%，各種神經網路都在 5.6% 到 7% 之間。卷積式的建構和全連接打平，參數卻少得多。全連接約 $10^7$ 個參數、5.6%；平滑係數的頻域建構約 $9\\times10^5$ 個參數、6%。\n\n旋轉完全隨機時，最近鄰錯了 80%，全連接 52%，平滑頻域建構 50%，是表中最好的，但所有方法都不理想。作者認為這需要完整的旋轉加平移不變性，而三維旋轉群不可交換，可能要更深的架構。\n\n## 結論\n\n這篇論文提出了兩條路。空間建構直接在圖上做局部連接和分群池化，參數降到 $O(n)$，但沒有權重共享；頻域建構用拉普拉斯的特徵向量當傅立葉基底，再用平滑的頻率係數換取空間上的局部性，參數可以降到 $O(1)$。\n\n頻域這條路的代價也一起留了下來。它要做特徵分解，每次轉換都乘一個 $n\\times n$ 的稠密矩陣，局部性也只是近似。三年後，ChebNet 把平滑的樣條換成拉普拉斯的多項式，保證局部，也不必特徵分解；再三個月，GCN 把多項式砍到一階。\n\n\n> [!TIP]\n> \n> 接下來兩步的簡化，可以參考我們的筆記：\n> \n> - **[[16.06] ChebNet：多項式就好](/blog/papers/graph-learning/chebnet)**\n> - **[[16.09] GCN：一個參數夠嗎？](/blog/papers/graph-learning/gcn)**\n\n\n論文裡那個從共變異數矩陣找回 CNN 的例子，到現在讀起來還是很有意思。卷積可以看成在某組基底下的對角運算，網格只是剛好提供了那組基底；在圖上，這組基底得自己去找。\n","imageSizes":{"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png":{"width":1210,"height":448},"/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png":{"width":908,"height":1208}}}