{"notes":[{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"多項式就好","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/chebnet.6954c188a205.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"空間上定義不出平移","text":"空間上定義不出平移","depth":3},{"id":"頻域上算不起","text":"頻域上算不起","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"用多項式換局部性","text":"用多項式換局部性","depth":3},{"id":"用-chebyshev-遞迴換速度","text":"用 Chebyshev 遞迴換速度","depth":3},{"id":"圖上的池化","text":"圖上的池化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"先在網格上驗證","text":"先在網格上驗證","depth":3},{"id":"三種濾波器的比較","text":"三種濾波器的比較","depth":3},{"id":"圖建得好不好","text":"圖建得好不好","depth":3},{"id":"後來的事","text":"後來的事","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"繞道頻域","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/spectral-networks.31f7482724ab.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"空間建構局部連接加分群","text":"空間建構：局部連接加分群","depth":3},{"id":"頻域建構拉普拉斯的特徵向量","text":"頻域建構：拉普拉斯的特徵向量","depth":3},{"id":"重新發現-cnn","text":"重新發現 CNN","depth":3},{"id":"平滑的係數局部的濾波器","text":"平滑的係數，局部的濾波器","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"次取樣的-mnist","text":"次取樣的 MNIST","depth":3},{"id":"球面上的-mnist","text":"球面上的 MNIST","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一個參數夠嗎？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/gcn.66a3d9542810.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"把圖當成正則項","text":"把圖當成正則項","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"從頻域一路簡化","text":"從頻域一路簡化","depth":3},{"id":"一層-gcn","text":"一層 GCN","depth":3},{"id":"兩層就夠","text":"兩層就夠","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"哪一種傳播最好","text":"哪一種傳播最好","depth":3},{"id":"疊深一點會怎樣","text":"疊深一點會怎樣","depth":3},{"id":"和-wl-演算法的關係","text":"和 WL 演算法的關係","depth":3},{"id":"四天後的一篇評論","text":"四天後的一篇評論","depth":3},{"id":"論文自己列的限制","text":"論文自己列的限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"CNN 在影像上好用，靠的是三件事。卷積核只看一小塊區域，同一組權重在整張圖上重複使用，一層一層疊上去還能組出越來越大的圖樣。\n\n這三件事都建立在網格上。像素有上下左右，「往右平移一格」有明確的意思。社群網路、腦部的連結圖、詞向量之間的相似度，資料都長在圖上，每個節點的鄰居數量不同，也沒有順序，平移這件事根本定義不出來。\n\nEPFL 的 Michaël Defferrard、Xavier Bresson 與 Pierre Vandergheynst 來自圖訊號處理的領域。Vandergheynst 早年和 Hammond 等人用 Chebyshev 多項式在圖上做小波，這篇把同一套工具帶進了 CNN。\n\n## 定義問題\n\n### 空間上定義不出平移\n\n要在圖上定義卷積，第一個想法是直接在空間上做，像影像那樣，拿一個固定大小的核去對齊每個節點的鄰居。作者指出，這條路得先解決「鄰居怎麼對齊」的問題；在圖上，從空間的角度，沒有唯一的數學定義能描述平移。\n\n另一條路是繞到頻域。卷積定理說，卷積是在傅立葉基底下會被對角化的線性運算。圖上也有傅立葉基底，就是拉普拉斯矩陣的特徵向量。\n\n### 頻域上算不起\n\n組合拉普拉斯 $L=D-W$ 或正規化拉普拉斯 $L=I_n-D^{-1/2}WD^{-1/2}$ 都是實對稱半正定矩陣，可以拆成 $L=U\\Lambda U^\\top$。$U$ 的每一行是一個「傅立葉模態」，特徵值 $\\lambda_l$ 就是它的頻率。訊號 $x\\in\\mathbb{R}^n$ 的圖傅立葉轉換是 $\\hat x=U^\\top x$。\n\n於是濾波可以寫成：\n\n$$\ny=g_\\theta(L)x=U\\,g_\\theta(\\Lambda)\\,U^\\top x\n$$\n\n最直接的做法是讓每個頻率的係數都自由學，$g_\\theta(\\Lambda)=\\mathrm{diag}(\\theta)$。這樣有兩個問題。濾波器在空間上不局部，可能一下子牽動整張圖；參數數量是 $O(n)$，跟節點數一樣多。\n\n就算把參數數量壓下來，計算還是很貴。2014 年 Bruna 等人的頻域網路用平滑的樣條函數參數化頻率係數，換取空間上的局部性。但每次前向、反向傳遞都要乘兩次 $U$，各是 $O(n^2)$；$U$ 本身要先對 $L$ 做特徵分解，還得存下一個 $n\\times n$ 的稠密矩陣。\n\n\n> [!TIP]\n> \n> Bruna 等人的空間與頻域兩種建構，可以參考我們的筆記：\n> \n> - **[[13.12] Spectral Networks：繞道頻域](/blog/papers/graph-learning/spectral-networks)**\n\n\n這篇要解決的，是一個在圖上**嚴格局部**、而且**算得快**的濾波器。\n\n## 解決問題\n\n### 用多項式換局部性\n\n作者把濾波器限制成 $\\Lambda$ 的多項式：\n\n$$\ng_\\theta(\\Lambda)=\\sum_{k=0}^{K-1}\\theta_k\\Lambda^k\n$$\n\n因為 $(U\\Lambda U^\\top)^k=U\\Lambda^kU^\\top$，這等於直接對拉普拉斯矩陣取多項式 $\\sum_k\\theta_kL^k$。$L$ 只在相鄰節點之間有非零值，$L^k$ 就只連得到 $k$ 步以內的節點；兩個節點的最短距離大於 $k$，$(L^k)_{ij}$ 就是 0。\n\n拿一條七個節點的路徑試試看，在正中間放一個 1，看 $L^k$ 會碰到哪些節點：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\n# 一條 7 個節點的路徑：0-1-2-3-4-5-6\nn = 7\nA = np.zeros((n, n))\nfor i in range(n - 1):\n    A[i, i + 1] = A[i + 1, i] = 1\nd = A.sum(1)\nL = np.eye(n) - A / np.sqrt(np.outer(d, d))   # 正規化拉普拉斯\n\ndelta = np.zeros(n)\ndelta[3] = 1                                   # 只在正中間的節點放一個 1\nfor k in range(4):\n    print(f\"L^{k} δ:\", (np.linalg.matrix_power(L, k) @ delta != 0).astype(int))\n```\n\n```text\nL^0 δ: [0 0 0 1 0 0 0]\nL^1 δ: [0 0 1 1 1 0 0]\nL^2 δ: [0 1 1 1 1 1 0]\nL^3 δ: [1 1 1 1 1 1 1]\n```\n\n每多一次方，影響範圍就往外擴一步。$K-1$ 階的多項式，影響範圍剛好是 $K-1$ 步以內，參數也只有 $K$ 個，和節點數無關。這和影像上 5×5 的卷積核只看 5×5 的範圍是同一件事。\n\n### 用 Chebyshev 遞迴換速度\n\n局部性有了，但如果還是先特徵分解、再在頻域裡乘，成本依舊是 $O(n^2)$。\n\n作者改用 Chebyshev 多項式 $T_k$ 當基底。它滿足一個穩定的遞迴關係：\n\n$$\nT_k(x)=2xT_{k-1}(x)-T_{k-2}(x),\\qquad T_0=1,\\quad T_1=x\n$$\n\nChebyshev 多項式定義在 $[-1,1]$，所以先把拉普拉斯縮放成 $\\tilde L=2L/\\lambda_{\\max}-I_n$，濾波器寫成：\n\n$$\ny=\\sum_{k=0}^{K-1}\\theta_kT_k(\\tilde L)\\,x\n$$\n\n令 $\\bar x_k=T_k(\\tilde L)x$，就有 $\\bar x_k=2\\tilde L\\bar x_{k-1}-\\bar x_{k-2}$。整個濾波只需要 $K$ 次稀疏矩陣乘向量，成本是 $O(K|E|)$。不用特徵分解，也不用存 $U$。\n\n接著上面的路徑，用兩種方法算同一個 $K=3$ 的濾波器，確認答案一樣：\n\n```python\ntheta = np.array([0.5, -1.0, 0.3])            # K = 3，三個 Chebyshev 係數\nx = np.random.default_rng(0).random(n)\n\n# 做法一：特徵分解，在頻域裡乘上濾波器\nlam, U = np.linalg.eigh(L)\nlam_max = lam.max()\nlam_s = 2 * lam / lam_max - 1                  # 把頻率縮放到 [-1, 1]\ng = theta[0] + theta[1] * lam_s + theta[2] * (2 * lam_s**2 - 1)   # T0, T1, T2\ny_spec = U @ (g * (U.T @ x))\n\n# 做法二：Chebyshev 遞迴，只做稀疏矩陣乘法\nL_s = 2 * L / lam_max - np.eye(n)\nx0, x1 = x, L_s @ x\nx2 = 2 * L_s @ x1 - x0\ny_rec = theta[0] * x0 + theta[1] * x1 + theta[2] * x2\n\nprint(\"頻域:\", y_spec)\nprint(\"遞迴:\", y_rec)\n```\n\n```text\n頻域: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\n遞迴: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\n```\n\n示範裡為了方便比對，$\\lambda_{\\max}$ 是用特徵分解求的；實際上只需要估最大特徵值，正規化拉普拉斯的特徵值也一定不超過 2。\n\n多個輸入、輸出通道時，第 $j$ 個輸出特徵圖是 $y_j=\\sum_i g_{\\theta_{ij}}(L)x_i$，每一對輸入輸出通道各有一組 $K$ 個係數。反向傳播同樣只需要 $K$ 次稀疏矩陣乘法。\n\n### 圖上的池化\n\nCNN 還需要池化。作者用 Graclus 多層分群演算法做粗化，每一層把相鄰的節點兩兩合併，節點數大約減半；配對時挑讓 normalized cut 最大的鄰居。\n\n粗化之後，節點的順序沒有任何意義，池化時得查表才知道誰和誰合併，又慢又難平行化。作者的辦法是補上一些不連任何邊的「假節點」，讓每個節點剛好有兩個子節點，排成一棵平衡二元樹，再依樹的結構重新排列節點。排好之後，圖上的池化就變成一維訊號的池化，相鄰的兩個數字合併成一個。\n\n[![原論文 Figure 1：從輸入的圖訊號開始，經過圖卷積層（卷積、非線性）與圖粗化（取樣、池化）提取特徵，再接全連接層做分類，輸出標籤。圖中也畫出頻域濾波器在不同頻率上的響應。](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)\n\n*原論文 Figure 1（Defferrard 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/1606.09375v3#page=2)。圖說未收錄。*\n\n整個網路就是圖卷積、非線性、粗化、池化的組合，最後接全連接層，和 LeNet 的骨架一樣。\n\n## 討論\n\n### 先在網格上驗證\n\n第一個實驗是 MNIST。把 28×28 的像素網格建成 8 近鄰圖，加上假節點後共 976 個節點、3,198 條邊。網路照 TensorFlow 教學裡的 LeNet-5 架構，影像版用 5×5 卷積，圖版用 $K=25$ 的濾波器，兩者看的範圍相當。\n\n| 模型 | 架構 | 準確率 |\n| --- | --- | ---: |\n| 一般 CNN | C32-P4-C64-P4-FC512 | 99.33% |\n| 圖 CNN | GC32-P4-GC64-P4-FC512 | 99.14% |\n\n差距很小。作者自己解釋，可能來自頻域濾波器的**等向性**，圖上的邊沒有上下左右的方向；這到底是缺點還是優點，要看任務，就像旋轉不變性有時候正是我們想要的。\n\n### 三種濾波器的比較\n\n同樣在 MNIST 上，比較三種頻域濾波器（$K=25$）：\n\n| 架構 | 自由係數 | 樣條（Bruna 等人） | Chebyshev |\n| --- | ---: | ---: | ---: |\n| GC10 | 95.75% | 97.26% | 97.48% |\n| GC32-P4-GC64-P4-FC512 | 96.28% | 97.15% | 99.14% |\n\n自由係數的濾波器不局部，參數又多，表現最差。Chebyshev 在兩種架構上都最好。\n\n速度方面，Bruna 等人的方法隨節點數以 $O(n^2)$ 成長，這篇是 $O(n)$。在 20NEWS 上把詞數從 2,000 增加到 12,000，舊方法處理一個批次的時間從幾十毫秒爬到 1.2 秒左右，Chebyshev 只增加到 150 毫秒上下。\n\n不過和一般 CNN 比，圖 CNN 還是慢了不少。一批 100 張 MNIST，CNN 在 GPU 上 31 毫秒，圖 CNN 要 200 毫秒；搬到 GPU 上的加速比倒是差不多，分別是 6.77 倍與 8 倍。\n\n### 圖建得好不好\n\n20NEWS 是文字分類，節點是 10,000 個常用詞，用 word2vec 向量建 16 近鄰圖。圖 CNN 的準確率是 68.26%，贏過全連接網路，但沒有贏過多項式 Naive Bayes 的 68.51%。\n\n作者花了一節談圖的品質。把 MNIST 的網格圖換成隨機圖，同一個五層架構從 99.14% 掉到 95.39%。卷積的前提是局部性、平穩性和組合性，資料所在的圖得滿足這些假設，卷積才有意義。\n\n### 後來的事\n\n三個月後，Kipf 和 Welling 的 GCN 以這篇為基礎，把多項式砍到一階，再把兩個係數綁成一個。在引用網路的半監督分類上，GCN 比 $K=2$、$K=3$ 的 Chebyshev 濾波器還準，Pubmed 上差了 5 個百分點左右。\n\nGCN 發表四天後，Ferenc Huszár 在部落格評論 GCN 時說，簡化太過頭會讓模型在影像網格上只剩一個固定的核；相較之下，ChebNet 的高階多項式雖然計算成本較高，看起來更有前景。[inFERENCe，2016-09-13](https://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/)\n\n\n> [!TIP]\n> \n> GCN 的簡化推導與那場討論，可以參考我們的筆記：\n> \n> - **[[16.09] GCN：一個參數夠嗎？](/blog/papers/graph-learning/gcn)**\n\n\n## 結論\n\nChebNet 把頻域圖卷積從理論變成可以訓練的網路。多項式讓濾波器只看 $K-1$ 步以內的鄰居，Chebyshev 遞迴讓計算只需要稀疏矩陣乘法，不必做特徵分解。在 MNIST 上，它和同架構的 CNN 只差 0.19 個百分點。\n\n$K$ 是這個方法最重要的超參數。它同時決定每層看多遠、要學幾個係數；在引用網路這類圖上，GCN 的實驗顯示 $K$ 小一點反而比較準，用之前值得先掃幾個值。\n\n頻域這條路後來大多走向 GCN 那樣的一階近似。不過拉普拉斯的特徵向量並沒有退場，幾年後的 Graph Transformer，又把它們撿回來當成節點的位置編碼。\n","imageSizes":{"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png":{"width":1144,"height":484}}}