{"notes":[{"id":"graph-learning/spectral-networks","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Spectral Networks","subtitle":"A Detour through the Spectral Domain","author":"Z. Yuan","paperDate":"2013-12-21","publishedOn":"2023-09-06","paperTitle":"Spectral Networks and Locally Connected Networks on Graphs","paperUrl":"https://arxiv.org/abs/1312.6203","bibliography":{"title":"Spectral Networks and Locally Connected Networks on Graphs","authors":["Joan Bruna","Wojciech Zaremba","Arthur Szlam","Yann LeCun"],"url":"https://arxiv.org/abs/1312.6203","pdfUrl":"https://arxiv.org/pdf/1312.6203v3","codeUrl":null,"preprint":{"id":"1312.6203","submittedOn":"2013-12-21","url":"https://arxiv.org/abs/1312.6203"},"publication":{"venue":"ICLR","year":2014,"kind":"conference","url":"https://iclr.cc/archive/2014/conference-proceedings/"},"citation":"@misc{graphlearningspectralnetworks2013,\n  title = {{Spectral Networks and Locally Connected Networks on Graphs}},\n  author = {Joan Bruna and Wojciech Zaremba and Arthur Szlam and Yann LeCun},\n  year = {2013},\n  eprint = {1312.6203},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1312.6203}\n}\n","citationPath":"/content/papers/citations/graph-learning/spectral-networks.ea286632c50d.bib"},"thumbnail":{"src":"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png","width":1210,"height":448},"locale":"en","path":"/content/papers/en/graph-learning/spectral-networks.cc53a07db581.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"spatial-construction-local-connections-and-clustering","text":"Spatial Construction: Local Connections and Clustering","depth":3},{"id":"spectral-construction-laplacian-eigenvectors","text":"Spectral Construction: Laplacian Eigenvectors","depth":3},{"id":"rediscovering-cnns","text":"Rediscovering CNNs","depth":3},{"id":"smooth-coefficients-local-filters","text":"Smooth Coefficients, Local Filters","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"subsampled-mnist","text":"Subsampled MNIST","depth":3},{"id":"mnist-on-a-sphere","text":"MNIST on a Sphere","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/chebnet","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/chebnet","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ChebNet","subtitle":"A Polynomial Will Do","author":"Z. Yuan","paperDate":"2016-06-30","publishedOn":"2023-09-16","world":"external/chebnet","paperTitle":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","paperUrl":"https://arxiv.org/abs/1606.09375","bibliography":{"title":"Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering","authors":["Michaël Defferrard","Xavier Bresson","Pierre Vandergheynst"],"url":"https://arxiv.org/abs/1606.09375","pdfUrl":"https://arxiv.org/pdf/1606.09375v3","codeUrl":"https://github.com/mdeff/cnn_graph","preprint":{"id":"1606.09375","submittedOn":"2016-06-30","url":"https://arxiv.org/abs/1606.09375"},"publication":{"venue":"NeurIPS","year":2016,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html"},"citation":"@misc{graphlearningchebnet2016,\n  title = {{Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering}},\n  author = {Michaël Defferrard and Xavier Bresson and Pierre Vandergheynst},\n  year = {2016},\n  eprint = {1606.09375},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1606.09375}\n}\n","citationPath":"/content/papers/citations/graph-learning/chebnet.52e0fac25946.bib"},"thumbnail":{"src":"/content/papers/assets/f924fc7fdf6829dd352c02c0062d56fef579382aee9d8de9a76c29aa312325ef.png","width":1144,"height":484},"locale":"en","path":"/content/papers/en/graph-learning/chebnet.6f3f6320bdd0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"translation-has-no-spatial-definition","text":"Translation Has No Spatial Definition","depth":3},{"id":"spectral-computation-is-too-expensive","text":"Spectral Computation Is Too Expensive","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"polynomials-buy-locality","text":"Polynomials Buy Locality","depth":3},{"id":"chebyshev-recursion-buys-speed","text":"Chebyshev Recursion Buys Speed","depth":3},{"id":"pooling-on-graphs","text":"Pooling on Graphs","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"first-validate-on-a-grid","text":"First Validate on a Grid","depth":3},{"id":"comparing-three-filters","text":"Comparing Three Filters","depth":3},{"id":"how-good-is-the-graph","text":"How Good Is the Graph?","depth":3},{"id":"what-came-next","text":"What Came Next","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"Is One Parameter Enough?","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"en","path":"/content/papers/en/graph-learning/gcn.61f7394131fa.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-graph-as-a-regularizer","text":"The Graph as a Regularizer","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"simplifying-from-the-spectral-domain","text":"Simplifying from the Spectral Domain","depth":3},{"id":"one-gcn-layer","text":"One GCN Layer","depth":3},{"id":"two-layers-are-enough","text":"Two Layers Are Enough","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-propagation-rule-works-best","text":"Which Propagation Rule Works Best?","depth":3},{"id":"what-happens-with-more-layers","text":"What Happens with More Layers?","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"a-review-four-days-later","text":"A Review Four Days Later","depth":3},{"id":"the-papers-own-limitations","text":"The Paper's Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"By late 2013, just over a year had passed since AlexNet's sweeping victory in image recognition. Joan Bruna, Wojciech Zaremba and Yann LeCun at New York University, together with Arthur Szlam at the City College of New York, wanted to know whether convolution could work on data that did not live on a grid.\n\nWeather-station networks, temperatures on 3D meshes, social networks and recommender systems all have some structure describing which things are close, but no regular up, down, left and right.\n\nThe paper proposes two ways to build CNNs on graphs. Three years later, ChebNet and GCN would simplify its spectral construction; the GCN paper explicitly builds on the spectral graph convolutions introduced here.\n\n## Defining the Problem\n\nThe authors first unpack why CNNs use so few parameters. On a grid, a CNN uses three structures at once:\n\n1. **Translation**: the same filter is reused at every position, giving weight sharing.\n2. **Distance**: a filter sees only a small neighborhood, with support much smaller than the full input.\n3. **Multiscale dyadic clustering**: grid locations can be merged in pairs, level by level, enabling strided convolution and pooling.\n\nFor an input with $n$ coordinates, a fully connected layer with $m$ outputs needs $n\\cdot m$ parameters, usually $O(n^2)$. Translation or local connectivity alone reduces this to $O(n)$ per feature map; together, they give $O(k\\cdot S)$, where $k$ is the number of feature maps and $S$ the filter size, independent of $n$.\n\nA graph has no translation and may have no useful coordinates. Given only a weighted graph $W$, the authors ask how much of these three structures can be recovered.\n\n## Solving the Problem\n\n### Spatial Construction: Local Connections and Clustering\n\nThe first approach works directly in space. Graph weights already define a neighborhood: for example, choose a threshold $\\delta$ and treat nodes with $W_{ij}>\\delta$ as neighbors. Connecting filters only to neighbors reduces the parameters from $O(n^2)$ to $O(S\\cdot n)$, where $S$ is the average neighborhood size.\n\nNext comes multiscale hierarchical clustering, dividing the nodes into $d_k$ groups at level $k$. Each layer first applies sparse filters connected only to neighbors, then pools within each group:\n\n$$\nx_{k+1,j}=L_k\\,h\\left(\\sum_{i=1}^{f_{k-1}}F_{k,i,j}\\,x_{k,i}\\right)\n$$\n\n$F_{k,i,j}$ is a sparse matrix with values only at neighboring positions, $h$ is a nonlinearity, and $L_k$ pools each group.\n\n[![Figure 2 of the paper: spatial construction at three resolutions. Gray nodes on the left form the original graph. Red regions show the first layer's local filtering and clustering, producing fewer red nodes in the middle; green regions repeat the process to produce the green nodes on the right. Each layer has fewer nodes and more feature dimensions.](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)](/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png)\n\n*Figure 2 from Bruna et al., taken from [**PDF page 4**](https://arxiv.org/pdf/1312.6203v3#page=4). The original caption is omitted.*\n\nMoving one layer to the right reduces spatial resolution and increases feature dimensions, much like a CNN pyramid.\n\nThis approach asks little of the graph, but has a fundamental limitation: filters at different locations learn separately, with no weight sharing. The parameter count is $O(n)$ rather than $O(1)$.\n\n### Spectral Construction: Laplacian Eigenvectors\n\nThe second approach takes a detour through the spectral domain. The authors use the combinatorial Laplacian $L=D-W$ to measure how smooth a signal is on the graph:\n\n$$\n\\left\\|\\nabla x\\right\\|_W^2=\\sum_i\\sum_jW_{ij}\\left[x(i)-x(j)\\right]^2\n$$\n\nThe smoothest vector is constant. Find the smoothest vector orthogonal to it, then the smoothest vector orthogonal to both, and continue: these are the eigenvectors $v_0, v_1, \\dots$ of $L$. Smaller eigenvalues correspond to smoother vectors. This is the same idea as the Fourier basis on a grid, where Laplacian eigenvectors are sinusoids.\n\nConvolution can then be defined as a diagonal operation in this basis:\n\n$$\nx_{k+1,j}=h\\left(V\\sum_{i=1}^{f_{k-1}}F_{k,i,j}V^\\top x_{k,i}\\right)\n$$\n\n$V$ is the eigenvector matrix, and $F_{k,i,j}$ is diagonal, multiplying each frequency by a coefficient. In practice, only the first $d$ low-frequency eigenvectors are retained, carrying the graph's smooth geometry. Each layer then has $f_{k-1}\\cdot f_k\\cdot d$ parameters, still $O(n)$.\n\nThe authors also spell out the difficulties. For most graphs, only the lowest-frequency eigenvectors are meaningful; both forward and backward passes require expensive multiplication by dense $V$ and $V^\\top$; and nonlinearities must be applied on the spatial side, so computation cannot stay in the spectral domain.\n\n### Rediscovering CNNs\n\nThe paper makes a lovely observation. Suppose the data covariance matrix supplies the graph weights. For natural images, translation invariance makes covariance depend only on the displacement between two pixels. Its eigenvectors are the Fourier basis, roughly ordered by frequency. The spectral operators $VF V^\\top$ are then diagonal in the Fourier basis, hence translation invariant: ordinary convolutions.\n\nIn other words, the method can recover a standard CNN by constructing a graph from data, without knowing in advance that images form a grid.\n\n### Smooth Coefficients, Local Filters\n\nTo reduce the parameters to $O(1)$, the authors borrow a duality from grids: a signal that decays rapidly in space is smooth in frequency. Conversely, a filter with a smooth frequency response tends to be concentrated in space.\n\nRather than learn a free coefficient for every frequency, they order eigenvectors along a line by eigenvalue, learn only a few control points, and interpolate between them with cubic splines:\n\n$$\n\\mathrm{diag}\\left(F_{k,i,j}\\right)=\\mathcal K\\,\\alpha_{k,i,j}\n$$\n\n$\\mathcal K$ is a fixed cubic-spline kernel, and $\\alpha$ contains the learned coefficients. The number of control points need not grow with $n$, giving $O(1)$ parameters.\n\nCompare the two on a path of 64 nodes. Place an impulse in the middle, filter it using either 64 free coefficients or smooth coefficients interpolated from 6 control points, and count how many nodes the response reaches:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(0)\n\nn = 64                                     # A path of 64 nodes\nW = np.zeros((n, n))\nfor i in range(n - 1):\n    W[i, i + 1] = W[i + 1, i] = 1\nL = np.diag(W.sum(1)) - W                  # Combinatorial Laplacian L = D - W\nlam, V = np.linalg.eigh(L)                 # Ordered from low to high frequency\n\nx = np.zeros(n)\nx[n // 2] = 1                              # Place an impulse at the center\n\ndef spread(y, ratio=0.05):\n    return int((np.abs(y) > ratio * np.abs(y).max()).sum())   # Count nodes with response above 5% of the maximum\n\nfrom scipy.interpolate import CubicSpline\n\nfree = rng.normal(size=n)                                   # One free coefficient per frequency, 64 in total\nknots = np.linspace(0, n - 1, 6)\nsmooth = CubicSpline(knots, rng.normal(size=6))(np.arange(n))  # Learn only 6 control points; interpolate the rest with a cubic spline\n\nfor name, g, params in [(\"Free coefficients\", free, n), (\"Smooth coefficients\", smooth, 6)]:\n    y = V @ (g * (V.T @ x))                                 # Filtering: V diag(g) V^T x\n    print(f\"{name}: {params:>2} parameters, response reaches {spread(y):>2} nodes\")\n```\n\n```text\nFree coefficients: 64 parameters, response reaches 60 nodes\nSmooth coefficients:  6 parameters, response reaches 15 nodes\n```\n\nThe filter with free coefficients reaches almost the whole path; the smooth filter concentrates around the impulse. Different random seeds show the same trend.\n\nSmoothness, however, only makes the filter more local; it gives no guarantee of support within a fixed number of hops. ChebNet would later close this gap with polynomials.\n\n## Discussion\n\n### Subsampled MNIST\n\nThe first experiment randomly samples 400 coordinates from 28×28 MNIST images. The points still lie in a plane, but no longer form a grid, so ordinary convolution cannot be used.\n\n| Method | Parameters | Error rate |\n| --- | ---: | ---: |\n| Nearest neighbor | — | 4.11% |\n| Fully connected 400-FC800-FC50-10 | $3.6\\times10^5$ | 1.8% |\n| Spatial construction, one layer | $7.2\\times10^4$ | 1.8% |\n| Spatial construction, two layers | $1.6\\times10^5$ | **1.3%** |\n| Spectral construction, free coefficients | $3.2\\times10^3$ | 2.6% |\n| Spectral construction, smooth coefficients ($q=32$) | $1.6\\times10^3$ | 2.3% |\n| Spectral construction, smooth coefficients ($q=20$, wider) | $5\\times10^3$ | 1.8% |\n\nThe two-layer spatial network is best, with 1.3% error. The spectral construction does somewhat worse but uses roughly one hundredth as many parameters as the fully connected model; smoothing reduces error from 2.6% to 2.3%, and widening reaches 1.8%.\n\nThe authors explain that MNIST digits consist of local, oriented strokes, requiring spatially concentrated measurements. Local connections provide locality directly; the spectral construction offers approximate locality through smoothing.\n\n[![Figure 6 of the paper: six plots show filter responses on 400 sampled points, with color encoding value. (a)(b) are spatial filters learned in two different clusters, colored only in small regions; (c)(d) are spectral filters with free coefficients, spread across the whole graph; (e)(f) are spectral filters with smooth coefficients, concentrated locally.](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)](/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png)\n\n*Figure 6 from Bruna et al., taken from [**PDF page 10**](https://arxiv.org/pdf/1312.6203v3#page=10). The original caption is omitted.*\n\nThe figure puts the three filters side by side. In (a)(b), two different clusters learn similar-looking features independently because their weights are not shared. In (c)(d), free spectral coefficients spread the response across the graph, which the authors call a fundamental limitation of Fourier analysis in describing local phenomena. Smoothing in (e)(f) concentrates the response again, while the same filter can detect similar features at different locations. That starts to look like convolution.\n\n### MNIST on a Sphere\n\nThe second experiment projects MNIST onto 4,096 random points on the unit sphere, then randomly rotates each image.\n\nWith small rotations, nearest-neighbor error rises to 19%, while the neural networks stay between 5.6% and 7%. Convolutional constructions match the fully connected model with far fewer parameters: roughly $10^7$ parameters and 5.6% error for the latter, versus about $9\\times10^5$ parameters and 6% for the smooth spectral construction.\n\nWith completely random rotations, nearest neighbor has 80% error, the fully connected model 52%, and the smooth spectral model 50%, the best in the table, though every method performs poorly. The authors suggest that full rotation and translation invariance is needed; the noncommutative 3D rotation group may call for a deeper architecture.\n\n## Conclusion\n\nThe paper opens two paths. Spatial construction uses local connections and clustered pooling directly on the graph, reducing parameters to $O(n)$ without weight sharing. Spectral construction uses Laplacian eigenvectors as a Fourier basis and trades smooth frequency coefficients for spatial locality, reducing parameters to $O(1)$.\n\nThe spectral path also leaves its costs behind: eigendecomposition, multiplication by a dense $n\\times n$ matrix at every transform, and only approximate locality. Three years later, ChebNet replaces smooth splines with Laplacian polynomials, guaranteeing locality and avoiding eigendecomposition; three months after that, GCN cuts the polynomial to first order.\n\n\n> [!TIP]\n> \n> For the next two steps of simplification, see our notes:\n> \n> - **[[16.06] ChebNet: A Polynomial Will Do](/blog/papers/graph-learning/chebnet)**\n> - **[[16.09] GCN: Is One Parameter Enough?](/blog/papers/graph-learning/gcn)**\n\n\nThe paper's example recovering CNNs from a covariance matrix is still fascinating to read. Convolution can be viewed as a diagonal operation in a particular basis; the grid happens to supply that basis. On a graph, we have to find it ourselves.\n","imageSizes":{"/content/papers/assets/81c18ce1b25b0675fa18b9922bab2a4316a1582ec2824475056e04b90cda9baa.png":{"width":1210,"height":448},"/content/papers/assets/4bf962fd0048720795737f14a2ce523576f6e02d4fb66c41a36bd5192d92ffed.png":{"width":908,"height":1208}}}