{"notes":[{"id":"action-recognition/st-gcn","category":"action-recognition","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ST-GCN","subtitle":"Centripetal, Centrifugal","author":"Z. Yuan","paperDate":"2018-01-23","publishedOn":"2023-12-11","world":"external/st-gcn","paperTitle":"Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition","paperUrl":"https://arxiv.org/abs/1801.07455","bibliography":{"title":"Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition","authors":["Sijie Yan","Yuanjun Xiong","Dahua Lin"],"url":"https://arxiv.org/abs/1801.07455","pdfUrl":"https://arxiv.org/pdf/1801.07455v2","codeUrl":"https://github.com/yysijie/st-gcn","preprint":{"id":"1801.07455","submittedOn":"2018-01-23","url":"https://arxiv.org/abs/1801.07455"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/12328"},"citation":"@misc{actionrecognitionstgcn2018,\n  title = {{Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition}},\n  author = {Sijie Yan and Yuanjun Xiong and Dahua Lin},\n  year = {2018},\n  eprint = {1801.07455},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07455}\n}\n","citationPath":"/content/papers/citations/action-recognition/st-gcn.d4a6b81c8471.bib"},"thumbnail":{"src":"/content/papers/assets/75fab9132b72202980c799f04d5fa51cf9749352d5b47ce0d19b571aa8a39acc.png","width":868,"height":292},"locale":"en","path":"/content/papers/en/action-recognition/st-gcn.4573ee2c89d7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-spatiotemporal-graph","text":"A Spatiotemporal Graph","depth":3},{"id":"who-gets-which-graph-weights","text":"Who Gets Which Graph Weights?","depth":3},{"id":"three-partitions","text":"Three Partitions","depth":3},{"id":"matrix-multiplication-implementation","text":"Matrix-Multiplication Implementation","depth":3},{"id":"edge-importance","text":"Edge Importance","depth":3},{"id":"the-full-network","text":"The Full Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"datasets","text":"Datasets","depth":3},{"id":"partition-ablations","text":"Partition Ablations","depth":3},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gcn","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"Is One Parameter Enough?","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"en","path":"/content/papers/en/graph-learning/gcn.61f7394131fa.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-graph-as-a-regularizer","text":"The Graph as a Regularizer","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"simplifying-from-the-spectral-domain","text":"Simplifying from the Spectral Domain","depth":3},{"id":"one-gcn-layer","text":"One GCN Layer","depth":3},{"id":"two-layers-are-enough","text":"Two Layers Are Enough","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-propagation-rule-works-best","text":"Which Propagation Rule Works Best?","depth":3},{"id":"what-happens-with-more-layers","text":"What Happens with More Layers?","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"a-review-four-days-later","text":"A Review Four Days Later","depth":3},{"id":"the-papers-own-limitations","text":"The Paper's Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]}],"text":"To recognize what someone is doing in a video, we can look at the images—or just the skeleton.\n\nA skeleton is a sequence of joint coordinates. A pose estimator marks the head, shoulders, elbows, wrists, hips, knees and ankles in each frame. A ten-second video becomes a sequence of hundreds of frames with a dozen or so points each. The data is compact and unaffected by backgrounds and lighting; for bodily actions such as dancing, waving or falling, the skeleton alone provides many clues.\n\nThe skeleton is itself a graph: joints are nodes, bones are edges. Sijie Yan, Yuanjun Xiong and Dahua Lin at the Chinese University of Hong Kong bring GCN onto this graph, handling space and time together. They call it ST-GCN.\n\n## Defining the Problem\n\nEarlier skeleton-based action recognition roughly followed two approaches.\n\nOne relied on handcrafted designs: divide the body into parts and manually define their relations, or compute features on joint trajectories. Changing datasets meant rewriting rules.\n\nThe other used RNNs or temporal convolutions, concatenating all joint coordinates in each frame into a long vector and processing it over time. This discards skeletal connectivity: the model must infer from data that the right elbow connects to the right wrist but not the left knee.\n\nThe desired model should use skeletal connections directly, learn spatial and temporal patterns together, and require no handcrafted body parts. GCN supplies a framework for graph convolution, but was designed for graphs such as citation networks. Applying it to skeletons raises another problem first.\n\n## Solving the Problem\n\n### A Spatiotemporal Graph\n\nFor a sequence of $T$ frames with $N$ joints each, the authors build an undirected spatiotemporal graph. Every joint in every frame is a node, totaling $T\\times N$; its features are coordinates plus pose-estimation confidence.\n\nEdges come in two kinds. Within a frame, joints follow natural bodily connections; between adjacent frames, the same joint is connected over time. Kinetics uses 18 2D joints estimated by OpenPose, while NTU-RGB+D uses 25 3D joints tracked by Kinect. Connectivity comes from the data, without manually grouping body parts.\n\n[![Figure 1 of the paper: a spatiotemporal graph of a skeleton sequence. Blue dots are joints; bones connect joints within a frame, while green edges connect the same joint across adjacent frames. The arrow at lower right indicates time.](/content/papers/assets/9e7851cb79bcf44e244d3b165ae301cb07adf86ffa6961d683bb9b743c4d2275.png)](/content/papers/assets/9e7851cb79bcf44e244d3b165ae301cb07adf86ffa6961d683bb9b743c4d2275.png)\n\n*Figure 1 from Yan et al., taken from [**PDF page 1**](https://arxiv.org/pdf/1801.07455v2#page=1). The original caption is omitted.*\n\nOn Kinetics, one video's input tensor is $(3, T, 18, 2)$. The 3 channels are $(x, y, \\text{confidence})$, 18 is the joint count, and 2 means taking at most the two highest-confidence people per video. Each video is repeated to fill $T=300$ frames.\n\n### Who Gets Which Graph Weights?\n\nReturn first to ordinary 2D convolution. For a 3×3 convolution, output at position $x$ is:\n\n$$\nf_{out}(x)=\\sum_{h=1}^{3}\\sum_{w=1}^{3}f_{in}\\left(p(x,h,w)\\right)\\cdot w(h,w)\n$$\n\nThe authors split this into two functions. **Sampling function** $p$ selects positions—the surrounding 9 pixels on an image. **Weight function** $w$ assigns weights to each position. A fixed image grid gives nine positions—upper left, directly above, upper right, and so on—each with its own weights, simply indexed from a $(c, 3, 3)$ tensor.\n\nOn a graph, sampling is easy: take neighbors within distance 1. Weight assignment is harder, because neighbors lack a fixed upper-left or directly-above order.\n\nThe solution is to partition neighbors into a fixed $K$ groups, label them, and share weights within each group:\n\n$$\nf_{out}(v_{ti})=\\sum_{v_{tj}\\in B(v_{ti})}\\frac{1}{Z_{ti}(v_{tj})}\\,f_{in}(v_{tj})\\cdot w\\left(l_{ti}(v_{tj})\\right)\n$$\n\n$l_{ti}$ maps each neighbor to a group, and $Z$ is the group's node count, balancing contributions across groups. In this view, a 3×3 convolution partitions nine pixels into nine groups of one.\n\nTime is simpler because it already has an order. Further indexing spatial groups by previous or following frames defines convolution over a spatiotemporal neighborhood. Temporal range $\\Gamma$ is the temporal kernel size, set to 9 in the paper.\n\n### Three Partitions\n\nThe partition determines what the model can distinguish. The authors try three:\n\n[![Figure 3 of the paper: four panels. (a) Input skeleton with a red dashed D=1 neighborhood. (b) Uni-labeling colors every neighbor green. (c) Distance partition colors self green and neighbors blue. (d) Spatial partition uses a black cross for the skeleton's center of gravity: self is green, neighbors nearer the center are blue, and farther neighbors yellow.](/content/papers/assets/75fab9132b72202980c799f04d5fa51cf9749352d5b47ce0d19b571aa8a39acc.png)](/content/papers/assets/75fab9132b72202980c799f04d5fa51cf9749352d5b47ce0d19b571aa8a39acc.png)\n\n*Figure 3 from Yan et al., taken from [**PDF page 5**](https://arxiv.org/pdf/1801.07455v2#page=5). The original caption is omitted.*\n\n- **Uni-labeling** (panel b): the whole neighborhood is one group, $K=1$. The authors identify this with Kipf and Welling's GCN propagation rule. Sharing weights across all neighbors amounts to averaging their features before an inner product, losing relative joint relationships.\n- **Distance partition** (panel c): self forms one group and distance-1 neighbors another, $K=2$. Separate weights can represent local differences such as self minus neighbors, including relative joint displacement.\n- **Spatial configuration partition** (panel d): $K=3$. Self forms one group; neighbors nearer the skeleton's center of gravity are **centripetal**; farther neighbors are **centrifugal**. The center is the mean joint coordinate in a frame, and $r_i$ is joint $i$'s average distance to it over the training set. The motivation is that limb movements roughly divide into inward contraction and outward extension.\n\nPartition a simplified six-joint skeleton by spatial configuration:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\n# Simplified skeleton: 0 neck, 1 right shoulder, 2 right elbow, 3 left shoulder, 4 left elbow, 5 pelvis\nnames = [\"neck\", \"right shoulder\", \"right elbow\", \"left shoulder\", \"left elbow\", \"pelvis\"]\nxy = np.array([[0.0, 1.0], [-0.4, 0.9], [-0.6, 0.5], [0.4, 0.9], [0.6, 0.5], [0.0, 0.2]])\nbones = [(0, 1), (1, 2), (0, 3), (3, 4), (0, 5)]\nV = len(xy)\nA = np.zeros((V, V))\nfor i, j in bones:\n    A[i, j] = A[j, i] = 1\n\ncenter = xy.mean(0)                          # Skeleton centroid: mean of all joint coordinates\nr = np.linalg.norm(xy - center, axis=1)      # Distance from each joint to the centroid\n\nA_root, A_in, A_out = np.eye(V), np.zeros((V, V)), np.zeros((V, V))\nfor i in range(V):\n    for j in np.nonzero(A[i])[0]:\n        if r[j] < r[i]:\n            A_in[i, j] = 1                   # Centripetal: neighbor is nearer the centroid than self\n        else:\n            A_out[i, j] = 1                  # Centrifugal: neighbor is farther from the centroid\n\nprint(\"Distance to centroid:\", r)\nfor i in (0, 1, 2):\n    print(f\"{names[i]}: centripetal {[names[j] for j in A_in[i].nonzero()[0]]}, \"\n          f\"centrifugal {[names[j] for j in A_out[i].nonzero()[0]]}\")\n```\n\n```text\nDistance to centroid: [0.33 0.46 0.62 0.46 0.62 0.47]\nneck: centripetal [], centrifugal ['right shoulder', 'left shoulder', 'pelvis']\nright shoulder: centripetal ['neck'], centrifugal ['right elbow']\nright elbow: centripetal ['right shoulder'], centrifugal []\n```\n\nThe example uses one frame; the paper averages distances over the training set. The right shoulder's two neighbors fall into different groups: toward the neck is centripetal, toward the elbow centrifugal. Uni-labeling would average them together.\n\n\n> [!TIP]\n> \n> When reading the official code, note a difference from the paper. The `spatial` strategy in `net/utils/graph.py` compares graph hop counts from each neighbor and self to a **designated central joint**—the neck for OpenPose's 18-point skeleton, the spine for NTU—rather than distances to the coordinate centroid. Neighbors at the same hop count join the self group.\n\n\n### Matrix-Multiplication Implementation\n\nThe implementation follows GCN. Under uni-labeling, one layer is:\n\n$$\nf_{out}=\\Lambda^{-\\frac12}(A+I)\\Lambda^{-\\frac12}f_{in}W\n$$\n\nFor multiple groups, split $A+I$ into $K$ matrices, $A+I=\\sum_jA_j$, normalize and weight each separately, then sum:\n\n$$\nf_{out}=\\sum_j\\Lambda_j^{-\\frac12}A_j\\Lambda_j^{-\\frac12}f_{in}W_j\n$$\n\nFor distance partitioning, $A_0=I$ and $A_1=A$. Some groups can have entirely zero rows—for example, the neck above has no centripetal neighbor—so normalization adds $\\alpha=0.001$ to degrees.\n\nUsing the same skeleton and an input tensor $(C, T, V)$, run one spatial graph convolution:\n\n```python\ndef normalize(Aj, alpha=0.001):\n    d = Aj.sum(1) + alpha                    # The paper adds alpha to handle all-zero rows\n    return Aj / np.sqrt(np.outer(d, d))\n\nC_in, C_out, T = 3, 8, 4\nx = np.random.default_rng(0).random((C_in, T, V))      # (channels, frames, joints)\nWs = [np.random.default_rng(k).normal(size=(C_out, C_in)) for k in range(3)]\n\nout = sum(np.einsum(\"oc,ctv,wv->otw\", Wk, x, normalize(Ak))   # Normalize and weight each group separately\n          for Wk, Ak in zip(Ws, [A_root, A_in, A_out]))\nprint(\"Input:\", x.shape, \"→ Output:\", out.shape)\n```\n\n```text\nInput: (3, 4, 6) → Output: (8, 4, 6)\n```\n\nA real ST-GCN unit follows this with a $1\\times\\Gamma$ temporal convolution, connecting the same joint across nearby frames.\n\n### Edge Importance\n\nA joint may belong to multiple body parts and have different importance within each. Each adjacency matrix in each layer is multiplied elementwise by a learnable mask $M$, initialized to all ones. The authors also suggest replacing this with data-dependent attention in future work.\n\n### The Full Network\n\nInput first passes through BatchNorm to align joint-coordinate scales, important because graph convolution shares weights over all nodes. Nine ST-GCN units follow, with output channels 64, 64, 64, 128, 128, 128, 256, 256, 256 and temporal kernel size 9. Each unit has a residual connection, followed by dropout 0.5. Units 4 and 7 use temporal stride 2 for pooling. Global pooling produces a 256-dimensional vector, then softmax classifies it.\n\nTraining uses SGD, learning rate 0.01, multiplied by 0.1 every 10 epochs. On Kinetics, two augmentations replace dropout: random affine transformations interpolated across frames to mimic camera movement, and random temporal crops during training, with full sequences at test time.\n\n## Discussion\n\n### Datasets\n\n**Kinetics** contains around 300,000 YouTube videos, 400 action classes and roughly 10 seconds per clip, with 240,000 training and 20,000 validation clips. Videos are resized to 340×256 at 30 FPS, and OpenPose extracts skeletons, the only model input.\n\n**NTU-RGB+D** is recorded in a laboratory: 40 subjects, three cameras recording simultaneously, around 56,000 clips and 60 classes. Kinect tracks 25 3D joints per person. X-Sub splits by subject, while X-View splits by camera viewpoint.\n\n### Partition Ablations\n\nTop-1 and top-5 accuracies on Kinetics:\n\n| Setting | Top-1 | Top-5 |\n| --- | ---: | ---: |\n| Temporal-convolution baseline (TCN) | 20.3% | 40.0% |\n| Sparse skeleton graph, unshared weights | 22.0% | 43.2% |\n| Uni-labeling | 19.3% | 37.4% |\n| Distance partition, weights tied with opposite signs | 23.9% | 44.9% |\n| Distance partition | 29.1% | 51.3% |\n| Spatial configuration partition | 29.9% | 52.2% |\n| Spatial configuration + edge importance | **30.7%** | **52.8%** |\n\nSeveral points stand out.\n\nFirst, uni-labeling—GCN-style propagation—scores below even the temporal baseline that ignores graph structure. The explanation is the loss of local difference information through averaging.\n\nSecond, separating self and neighbors jumps from 19.3% to 29.1%. Even tying the two groups' weights to differ only by sign beats uni-labeling, which the authors take as evidence for partitioning itself.\n\nThird, three spatial groups improve further on two, and edge importance adds nearly another percentage point.\n\nThe temporal baseline is actually equivalent to spatiotemporal graph convolution on a fully connected joint graph with unshared weights. ST-GCN differs in using sparse natural connections and shared weights.\n\n### Comparison with Other Methods\n\n| Method | Kinetics Top-1 | NTU X-Sub | NTU X-View |\n| --- | ---: | ---: | ---: |\n| Deep LSTM | 16.4% | 60.7% | 67.3% |\n| Temporal ConvNet | 20.3% | 74.3% | 83.1% |\n| C-CNN + MTLN | — | 79.6% | 84.8% |\n| **ST-GCN** | **30.7%** | **81.5%** | **88.3%** |\n\nST-GCN is the best skeleton-based method of the time on both datasets.\n\nOn Kinetics, skeletons remain far behind images: RGB reaches 57.0% and optical flow 49.5%. Many Kinetics actions, the authors explain, require seeing held objects or surroundings. On a selected subset of 30 classes closely tied to bodily movement, ST-GCN's mean class accuracy is 72.4%, versus 70.4% for RGB and 72.8% for optical flow—roughly comparable.\n\nCombined with an RGB two-stream model, ST-GCN helps more than optical flow. Without ImageNet pretraining, RGB alone reaches 70.3%, adding optical flow 71.1%, adding ST-GCN 71.2%, and using all three 71.7%.\n\n## Conclusion\n\nST-GCN builds skeleton sequences into a spatiotemporal graph and convolves over it using GCN's form. Neighbor partitioning is its most important design: without it, the model cannot beat a graph-free baseline; splitting into two groups adds nearly 10 percentage points.\n\nThis also gives a practical answer to GCN's collapse into an averaging kernel on image grids. On graphs with spatial structure, separating weights by neighbors' roles makes a large difference.\n\n\n> [!TIP]\n> \n> For GCN's propagation rule and why it shares weights across all neighbors, see our note:\n> \n> - **[[16.09] GCN: Is One Parameter Enough?](/blog/papers/graph-learning/gcn)**\n\n\nMany later skeleton action-recognition models build on ST-GCN; [2s-AGCN](https://arxiv.org/abs/1805.07694), for example, learns a data-dependent adjacency matrix. When implementing your own, first check how many joints your pose estimator outputs and which joint is the center. The partition depends on it.\n","imageSizes":{"/content/papers/assets/9e7851cb79bcf44e244d3b165ae301cb07adf86ffa6961d683bb9b743c4d2275.png":{"width":454,"height":364},"/content/papers/assets/75fab9132b72202980c799f04d5fa51cf9749352d5b47ce0d19b571aa8a39acc.png":{"width":868,"height":292}}}