{"notes":[{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"A Polynomial Will Do","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"en","path":"/content/papers/en/graph-learning/chebnet.6f3f6320bdd0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"translation-has-no-spatial-definition","text":"Translation Has No Spatial Definition","depth":3},{"id":"spectral-computation-is-too-expensive","text":"Spectral Computation Is Too Expensive","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"polynomials-buy-locality","text":"Polynomials Buy Locality","depth":3},{"id":"chebyshev-recursion-buys-speed","text":"Chebyshev Recursion Buys Speed","depth":3},{"id":"pooling-on-graphs","text":"Pooling on Graphs","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"first-validate-on-a-grid","text":"First Validate on a Grid","depth":3},{"id":"comparing-three-filters","text":"Comparing Three Filters","depth":3},{"id":"how-good-is-the-graph","text":"How Good Is the Graph?","depth":3},{"id":"what-came-next","text":"What Came Next","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"A Detour through the Spectral Domain","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"en","path":"/content/papers/en/graph-learning/spectral-networks.cc53a07db581.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"spatial-construction-local-connections-and-clustering","text":"Spatial Construction: Local Connections and Clustering","depth":3},{"id":"spectral-construction-laplacian-eigenvectors","text":"Spectral Construction: Laplacian Eigenvectors","depth":3},{"id":"rediscovering-cnns","text":"Rediscovering CNNs","depth":3},{"id":"smooth-coefficients-local-filters","text":"Smooth Coefficients, Local Filters","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"subsampled-mnist","text":"Subsampled MNIST","depth":3},{"id":"mnist-on-a-sphere","text":"MNIST on a Sphere","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"Is One Parameter Enough?","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"en","path":"/content/papers/en/graph-learning/gcn.61f7394131fa.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-graph-as-a-regularizer","text":"The Graph as a Regularizer","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"simplifying-from-the-spectral-domain","text":"Simplifying from the Spectral Domain","depth":3},{"id":"one-gcn-layer","text":"One GCN Layer","depth":3},{"id":"two-layers-are-enough","text":"Two Layers Are Enough","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-propagation-rule-works-best","text":"Which Propagation Rule Works Best?","depth":3},{"id":"what-happens-with-more-layers","text":"What Happens with More Layers?","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"a-review-four-days-later","text":"A Review Four Days Later","depth":3},{"id":"the-papers-own-limitations","text":"The Paper's Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"CNNs work well on images for three reasons: a kernel sees only a small region, the same weights are reused across the image, and stacking layers builds ever larger patterns.\n\nAll three rely on a grid. Pixels have up, down, left and right, and “shift one position to the right” has a clear meaning. Social networks, brain-connectivity graphs and word-vector similarities live on graphs instead. Neighborhood sizes vary, neighbors have no order, and translation cannot even be defined in the same way.\n\nMichaël Defferrard, Xavier Bresson and Pierre Vandergheynst at EPFL come from graph signal processing. Vandergheynst had previously used Chebyshev polynomials for graph wavelets with Hammond and colleagues; this paper brings the same tool into CNNs.\n\n## Defining the Problem\n\n### Translation Has No Spatial Definition\n\nThe first idea for graph convolution is to work directly in space: align a fixed-size kernel with each node's neighbors, as on an image. The authors point out that this first requires a rule for aligning neighbors. From a spatial perspective, a graph has no unique mathematical definition of translation.\n\nThe other path goes through the spectral domain. The convolution theorem says convolution is a linear operation diagonalized in the Fourier basis. A graph also has a Fourier basis: the eigenvectors of its Laplacian.\n\n### Spectral Computation Is Too Expensive\n\nBoth the combinatorial Laplacian $L=D-W$ and the normalized Laplacian $L=I_n-D^{-1/2}WD^{-1/2}$ are real, symmetric, positive semidefinite matrices, admitting $L=U\\Lambda U^\\top$. Each column of $U$ is a Fourier mode, with eigenvalue $\\lambda_l$ as its frequency. The graph Fourier transform of a signal $x\\in\\mathbb{R}^n$ is $\\hat x=U^\\top x$.\n\nFiltering can therefore be written as:\n\n$$\ny=g_\\theta(L)x=U\\,g_\\theta(\\Lambda)\\,U^\\top x\n$$\n\nThe most direct approach learns one free coefficient per frequency, $g_\\theta(\\Lambda)=\\mathrm{diag}(\\theta)$. This has two problems: the filter is spatially nonlocal and can affect the whole graph at once, and it has $O(n)$ parameters, as many as there are nodes.\n\nEven reducing the parameter count leaves computation expensive. Bruna et al.'s 2014 spectral networks parameterize frequency coefficients with smooth splines to obtain spatial locality. But every forward or backward pass still needs two multiplications by $U$, each costing $O(n^2)$; obtaining $U$ also requires eigendecomposing $L$ and storing a dense $n\\times n$ matrix.\n\n\n> [!TIP]\n> \n> For Bruna et al.'s spatial and spectral constructions, see our note:\n> \n> - **[[13.12] Spectral Networks: A Detour through the Spectral Domain](/blog/papers/graph-learning/spectral-networks)**\n\n\nThis paper seeks a graph filter that is both **strictly local** and **fast to compute**.\n\n## Solving the Problem\n\n### Polynomials Buy Locality\n\nThe authors restrict the filter to a polynomial in $\\Lambda$:\n\n$$\ng_\\theta(\\Lambda)=\\sum_{k=0}^{K-1}\\theta_k\\Lambda^k\n$$\n\nSince $(U\\Lambda U^\\top)^k=U\\Lambda^kU^\\top$, this is a polynomial directly in the Laplacian, $\\sum_k\\theta_kL^k$. $L$ has nonzero entries only between adjacent nodes, so $L^k$ reaches only nodes within $k$ hops; if two nodes have shortest-path distance greater than $k$, then $(L^k)_{ij}=0$.\n\nTry a path of seven nodes. Place a 1 in the middle and see which nodes $L^k$ reaches:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\n# A seven-node path: 0-1-2-3-4-5-6\nn = 7\nA = np.zeros((n, n))\nfor i in range(n - 1):\n    A[i, i + 1] = A[i + 1, i] = 1\nd = A.sum(1)\nL = np.eye(n) - A / np.sqrt(np.outer(d, d))   # Normalized Laplacian\n\ndelta = np.zeros(n)\ndelta[3] = 1                                   # Place a 1 only at the middle node\nfor k in range(4):\n    print(f\"L^{k} δ:\", (np.linalg.matrix_power(L, k) @ delta != 0).astype(int))\n```\n\n```text\nL^0 δ: [0 0 0 1 0 0 0]\nL^1 δ: [0 0 1 1 1 0 0]\nL^2 δ: [0 1 1 1 1 1 0]\nL^3 δ: [1 1 1 1 1 1 1]\n```\n\nEach extra power extends the reach by one hop. A degree-$K-1$ polynomial has support within $K-1$ hops and only $K$ parameters, independent of the number of nodes. This is the graph counterpart of a 5×5 image kernel seeing only a 5×5 region.\n\n### Chebyshev Recursion Buys Speed\n\nLocality is solved, but eigendecomposing first and multiplying in the spectral domain would still cost $O(n^2)$.\n\nThe authors instead use Chebyshev polynomials $T_k$ as the basis. They satisfy a stable recurrence:\n\n$$\nT_k(x)=2xT_{k-1}(x)-T_{k-2}(x),\\qquad T_0=1,\\quad T_1=x\n$$\n\nChebyshev polynomials are defined on $[-1,1]$, so the Laplacian is first rescaled to $\\tilde L=2L/\\lambda_{\\max}-I_n$, giving the filter:\n\n$$\ny=\\sum_{k=0}^{K-1}\\theta_kT_k(\\tilde L)\\,x\n$$\n\nSet $\\bar x_k=T_k(\\tilde L)x$, and $\\bar x_k=2\\tilde L\\bar x_{k-1}-\\bar x_{k-2}$. Filtering needs only $K$ sparse matrix–vector multiplications, at $O(K|E|)$ cost. No eigendecomposition and no stored $U$.\n\nContinuing with the same path, compute the same $K=3$ filter in two ways and check that they agree:\n\n```python\ntheta = np.array([0.5, -1.0, 0.3])            # K = 3, three Chebyshev coefficients\nx = np.random.default_rng(0).random(n)\n\n# Method 1: eigendecompose, then multiply by the spectral filter\nlam, U = np.linalg.eigh(L)\nlam_max = lam.max()\nlam_s = 2 * lam / lam_max - 1                  # Rescale frequencies to [-1, 1]\ng = theta[0] + theta[1] * lam_s + theta[2] * (2 * lam_s**2 - 1)   # T0, T1, T2\ny_spec = U @ (g * (U.T @ x))\n\n# Method 2: Chebyshev recursion, using only sparse matrix multiplication\nL_s = 2 * L / lam_max - np.eye(n)\nx0, x1 = x, L_s @ x\nx2 = 2 * L_s @ x1 - x0\ny_rec = theta[0] * x0 + theta[1] * x1 + theta[2] * x2\n\nprint(\"Spectral:\", y_spec)\nprint(\"Recursive:\", y_rec)\n```\n\n```text\nSpectral: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\nRecursive: [0.52 0.65 0.42 0.61 1.01 1.43 1.12]\n```\n\nFor easy comparison, the example obtains $\\lambda_{\\max}$ by eigendecomposition. In practice, only an estimate of the largest eigenvalue is needed; normalized-Laplacian eigenvalues are always at most 2.\n\nWith multiple input and output channels, output feature map $j$ is $y_j=\\sum_i g_{\\theta_{ij}}(L)x_i$, with a separate set of $K$ coefficients per input–output channel pair. Backpropagation likewise needs only $K$ sparse matrix multiplications.\n\n### Pooling on Graphs\n\nA CNN also needs pooling. The authors coarsen the graph using Graclus multilevel clustering, merging neighboring nodes in pairs at each level to roughly halve the node count; pairing selects the neighbor that maximizes the normalized-cut criterion.\n\nAfter coarsening, node order has no meaning. Pooling would need lookup tables to identify pairs, making it slow and hard to parallelize. The authors add disconnected “fake nodes” so every node has exactly two children in a balanced binary tree, then reorder nodes according to that tree. Graph pooling becomes pooling a one-dimensional signal, merging adjacent pairs of values.\n\n[![Figure 1 of the paper: an input graph signal passes through graph-convolution layers (convolution and nonlinearity) and graph coarsening (sampling and pooling) to extract features, then fully connected layers classify it and produce labels. The figure also shows spectral-filter responses at different frequencies.](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)](/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png)\n\n*Figure 1 from Defferrard et al., taken from [**PDF page 2**](https://arxiv.org/pdf/1606.09375v3#page=2). The original caption is omitted.*\n\nThe whole network combines graph convolution, nonlinearity, coarsening and pooling, followed by fully connected layers—the same skeleton as LeNet.\n\n## Discussion\n\n### First Validate on a Grid\n\nThe first experiment uses MNIST. The 28×28 pixel grid becomes an 8-nearest-neighbor graph, with 976 nodes and 3,198 edges after adding fake nodes. The network follows the LeNet-5 architecture in the TensorFlow tutorial: the image version uses 5×5 convolutions, while the graph version uses $K=25$ filters, covering comparable regions.\n\n| Model | Architecture | Accuracy |\n| --- | --- | ---: |\n| Ordinary CNN | C32-P4-C64-P4-FC512 | 99.33% |\n| Graph CNN | GC32-P4-GC64-P4-FC512 | 99.14% |\n\nThe gap is small. The authors suggest **isotropy** of spectral filters as a possible cause: graph edges carry no up, down, left or right. Whether this is a weakness or a strength depends on the task, just as rotation invariance can sometimes be desirable.\n\n### Comparing Three Filters\n\nThree spectral filters are compared on MNIST with $K=25$:\n\n| Architecture | Free coefficients | Splines (Bruna et al.) | Chebyshev |\n| --- | ---: | ---: | ---: |\n| GC10 | 95.75% | 97.26% | 97.48% |\n| GC32-P4-GC64-P4-FC512 | 96.28% | 97.15% | 99.14% |\n\nFree-coefficient filters are nonlocal, have many parameters and perform worst. Chebyshev is best with both architectures.\n\nBruna et al.'s runtime grows as $O(n^2)$ with the number of nodes, versus $O(n)$ here. On 20NEWS, increasing the vocabulary from 2,000 to 12,000 takes the old method's batch time from tens of milliseconds to around 1.2 seconds; Chebyshev rises only to roughly 150 milliseconds.\n\nGraph CNNs remain considerably slower than ordinary CNNs, however. On a GPU, a batch of 100 MNIST images takes 31 milliseconds for the CNN and 200 milliseconds for the graph CNN. Their GPU speedups are similar: 6.77× and 8×, respectively.\n\n### How Good Is the Graph?\n\n20NEWS is a text-classification task. Its nodes are 10,000 common words, with a 16-nearest-neighbor graph built from word2vec vectors. The graph CNN reaches 68.26% accuracy, beating the fully connected network but falling short of multinomial Naive Bayes at 68.51%.\n\nThe authors devote a section to graph quality. Replacing the MNIST grid graph with a random graph drops the same five-layer architecture from 99.14% to 95.39%. Convolution assumes locality, stationarity and compositionality; the data graph must support these assumptions for convolution to be meaningful.\n\n### What Came Next\n\nThree months later, Kipf and Welling build GCN on this paper, cutting the polynomial to first order and tying two coefficients into one. For semi-supervised classification on citation networks, GCN is more accurate than Chebyshev filters with $K=2$ or $K=3$, by around 5 percentage points on Pubmed.\n\nFour days after GCN appears, Ferenc Huszár writes in a blog review that excessive simplification leaves the model with a fixed kernel on an image grid. By comparison, ChebNet's higher-order polynomials look more promising, despite their greater computational cost. [inFERENCe, 2016-09-13](https://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/)\n\n\n> [!TIP]\n> \n> For GCN's simplified derivation and that discussion, see our note:\n> \n> - **[[16.09] GCN: Is One Parameter Enough?](/blog/papers/graph-learning/gcn)**\n\n\n## Conclusion\n\nChebNet turns spectral graph convolution from theory into a trainable network. Polynomials restrict a filter to neighbors within $K-1$ hops; Chebyshev recursion reduces computation to sparse matrix multiplication without eigendecomposition. On MNIST, it finishes only 0.19 percentage points behind a CNN with the same architecture.\n\n$K$ is the method's most important hyperparameter, determining both each layer's reach and the number of coefficients learned. On graphs such as citation networks, the GCN experiments show smaller $K$ can be more accurate, so it is worth sweeping a few values first.\n\nMuch of the spectral path later moves toward first-order approximations like GCN. Laplacian eigenvectors stay in the picture, though: a few years later, Graph Transformers pick them up again as node positional encodings.\n","imageSizes":{"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png":{"width":1144,"height":484}}}