{"notes":[{"id":"graph-learning/gat","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GAT","subtitle":"Some Neighbors Matter More","author":"Z. Yuan","paperDate":"2017-10-30","publishedOn":"2023-11-10","world":"external/gat","paperTitle":"Graph Attention Networks","paperUrl":"https://arxiv.org/abs/1710.10903","bibliography":{"title":"Graph Attention Networks","authors":["Petar Veličković","Guillem Cucurull","Arantxa Casanova","Adriana Romero","Pietro Liò","Yoshua Bengio"],"url":"https://arxiv.org/abs/1710.10903","pdfUrl":"https://arxiv.org/pdf/1710.10903v3","codeUrl":"https://github.com/PetarV-/GAT","preprint":{"id":"1710.10903","submittedOn":"2017-10-30","url":"https://arxiv.org/abs/1710.10903"},"publication":{"venue":"ICLR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1710.10903"},"citation":"@misc{graphlearninggat2017,\n  title = {{Graph Attention Networks}},\n  author = {Petar Veličković and Guillem Cucurull and Arantxa Casanova and Adriana Romero and Pietro Liò and Yoshua Bengio},\n  year = {2017},\n  eprint = {1710.10903},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1710.10903}\n}\n","citationPath":"/content/papers/citations/graph-learning/gat.402fa57986a5.bib"},"thumbnail":{"src":"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png","width":1124,"height":540},"locale":"en","path":"/content/papers/en/graph-learning/gat.8ad15224e4f4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"one-attention-head","text":"One Attention Head","depth":3},{"id":"multiple-heads","text":"Multiple Heads","depth":3},{"id":"cost-and-properties","text":"Cost and Properties","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"what-did-it-learn","text":"What Did It Learn?","depth":3},{"id":"the-authors-own-limitations","text":"The Authors' Own Limitations","depth":3},{"id":"static-attention","text":"Static Attention","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gcn","relation":"referenced","topic":""},{"id":"graph-learning/graphsage","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"Is One Parameter Enough?","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"en","path":"/content/papers/en/graph-learning/gcn.61f7394131fa.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-graph-as-a-regularizer","text":"The Graph as a Regularizer","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"simplifying-from-the-spectral-domain","text":"Simplifying from the Spectral Domain","depth":3},{"id":"one-gcn-layer","text":"One GCN Layer","depth":3},{"id":"two-layers-are-enough","text":"Two Layers Are Enough","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-propagation-rule-works-best","text":"Which Propagation Rule Works Best?","depth":3},{"id":"what-happens-with-more-layers","text":"What Happens with More Layers?","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"a-review-four-days-later","text":"A Review Four Days Later","depth":3},{"id":"the-papers-own-limitations","text":"The Paper's Own Limitations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]},{"id":"graph-learning/graphsage","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphSAGE","subtitle":"Sampling the Roll Call","author":"Z. Yuan","paperDate":"2017-06-07","publishedOn":"2023-10-21","world":"external/graphsage","paperTitle":"Inductive Representation Learning on Large Graphs","paperUrl":"https://arxiv.org/abs/1706.02216","bibliography":{"title":"Inductive Representation Learning on Large Graphs","authors":["William L. Hamilton","Rex Ying","Jure Leskovec"],"url":"https://arxiv.org/abs/1706.02216","pdfUrl":"https://arxiv.org/pdf/1706.02216v4","codeUrl":"https://github.com/williamleif/GraphSAGE","preprint":{"id":"1706.02216","submittedOn":"2017-06-07","url":"https://arxiv.org/abs/1706.02216"},"publication":{"venue":"NeurIPS","year":2017,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html"},"citation":"@misc{graphlearninggraphsage2017,\n  title = {{Inductive Representation Learning on Large Graphs}},\n  author = {William L. Hamilton and Rex Ying and Jure Leskovec},\n  year = {2017},\n  eprint = {1706.02216},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.02216}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphsage.1a0b3fd6e8b8.bib"},"thumbnail":{"src":"/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png","width":1090,"height":384},"locale":"en","path":"/content/papers/en/graph-learning/graphsage.f6ad2435eb15.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"transductive-and-inductive","text":"Transductive and Inductive","depth":3},{"id":"too-many-neighbors","text":"Too Many Neighbors","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-algorithm","text":"The Algorithm","depth":3},{"id":"sample-the-neighbors","text":"Sample the Neighbors","depth":3},{"id":"three-aggregators","text":"Three Aggregators","depth":3},{"id":"learning-without-labels","text":"Learning without Labels","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"three-datasets","text":"Three Datasets","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"speed-and-sampling-tradeoffs","text":"Speed and Sampling Tradeoffs","depth":3},{"id":"what-came-next","text":"What Came Next","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gat","relation":"citing","topic":""}]}],"text":"After attention proves itself in machine translation, the Transformer goes all in during June 2017, removing RNNs entirely.\n\nAttention does something simple: each position looks at others, decides how much to take based on similarity, and combines their values. This is almost the same as aggregation in graph neural networks; the difference is where the weights come from.\n\nGCN derives them from degrees: the share node $i$ receives from neighbor $j$ is fixed at $1/\\sqrt{\\tilde d_i\\tilde d_j}$. Petar Veličković at Cambridge, together with Bengio and colleagues, wants features to determine these shares. The result is GAT.\n\n## Defining the Problem\n\nThe authors lay out the difficulties of the two existing approaches.\n\nSpectral methods rely on Laplacian eigendecomposition, tying learned filters to that graph's eigenvectors. A graph with a different structure cannot directly reuse the model. GCN simplifies filtering to one-hop neighbors and is faster, but each neighbor's share within a neighborhood still depends only on degrees.\n\nSpatial methods must handle varying neighborhood sizes. GraphSAGE samples a fixed number per node, so inference does not see the full neighborhood. Its best-performing LSTM aggregator also assumes an order, forcing random neighbor shuffling each time.\n\nThe desired layer should parallelize, handle nodes of different degrees, assign different importance within one neighborhood, and apply directly to unseen graphs without knowing the whole graph structure in advance.\n\n## Solving the Problem\n\n### One Attention Head\n\nThe input is node features $h_i\\in\\mathbb{R}^F$. First, all nodes share a linear transformation $W\\in\\mathbb{R}^{F'\\times F}$. Then compute a score for each pair:\n\n$$\ne_{ij}=a\\left(Wh_i,\\ Wh_j\\right)\n$$\n\n$e_{ij}$ expresses the importance of node $j$'s features to node $i$. In the most general form, every node could see all nodes, discarding the graph structure. The authors instead use **masked attention**, computing only over neighbors $\\mathcal N_i$, including self, then normalizing with softmax:\n\n$$\n\\alpha_{ij}=\\mathrm{softmax}_j(e_{ij})=\\frac{\\exp(e_{ij})}{\\sum_{k\\in\\mathcal N_i}\\exp(e_{ik})}\n$$\n\nAttention mechanism $a$ is a single feedforward layer with weight vector $a\\in\\mathbb{R}^{2F'}$, followed by LeakyReLU with negative slope 0.2. Expanded:\n\n$$\n\\alpha_{ij}=\\frac{\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_j\\right]\\right)\\right)}{\\sum_{k\\in\\mathcal N_i}\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_k\\right]\\right)\\right)}\n$$\n\n$\\|$ denotes concatenation. These coefficients finally weight the neighbors: $h'_i=\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha_{ij}Wh_j\\right)$.\n\nRun one head on a five-node graph. Split $a^\\top[Wh_i\\|Wh_j]$ into a half depending only on $i$ and a half depending only on $j$, computing the whole score matrix at once:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(1)\n\n# 5 nodes: 0-1, 0-2, 1-2, 2-3, 3-4, with self-loops\nA = np.eye(5)\nfor i, j in [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]:\n    A[i, j] = A[j, i] = 1\n\nh = rng.normal(size=(5, 3))          # 3 features per node\nW = rng.normal(size=(3, 2))          # Shared linear transformation to 2 output dimensions\na = rng.normal(size=4)               # Attention vector of length 2F' = 4\n\ndef leaky_relu(z, slope=0.2):\n    return np.where(z > 0, z, slope * z)\n\ndef gat_head(h, W, a):\n    Wh = h @ W                                             # (5, 2)\n    e = leaky_relu((Wh @ a[:2])[:, None] + (Wh @ a[2:])[None, :])   # a^T [Wh_i || Wh_j], (5, 5)\n    e = np.where(A > 0, e, -np.inf)                        # Mask: retain only neighbors\n    alpha = np.exp(e - e.max(1, keepdims=True))\n    alpha /= alpha.sum(1, keepdims=True)                   # Row-wise softmax over neighbors\n    return alpha, alpha @ Wh\n\nalpha, out = gat_head(h, W, a)\nprint(\"Attention coefficients:\")\nprint(alpha)\nprint(\"Output:\", out.shape)\n```\n\n```text\nAttention coefficients:\n[[0.52 0.26 0.23 0.   0.  ]\n [0.52 0.26 0.22 0.   0.  ]\n [0.45 0.23 0.19 0.14 0.  ]\n [0.   0.   0.51 0.26 0.23]\n [0.   0.   0.   0.58 0.42]]\nOutput: (5, 2)\n```\n\nEvery row sums to 1, with zeros at non-neighbor positions. Nodes 0 and 1 both have three neighbors, to which GCN would assign similar shares. Here features determine the shares, and node 0 receives the largest.\n\n### Multiple Heads\n\nAs with the Transformer, multiple attention heads stabilize training. Each of $K$ heads has its own $W^k$ and $a^k$. Intermediate layers concatenate their independently computed outputs to dimension $KF'$. At the final layer, concatenation would no longer be useful, so outputs are averaged before softmax or sigmoid:\n\n$$\nh'_i=\\Big\\|_{k=1}^{K}\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right),\\qquad\nh'_i=\\sigma\\left(\\frac1K\\sum_{k=1}^K\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right)\n$$\n\nContinuing the example, use three sets of weights and compare the shapes of the two combinations:\n\n```python\nheads = [gat_head(h, rng.normal(size=(3, 2)), rng.normal(size=4))[1] for _ in range(3)]\nhidden = np.concatenate([np.where(o > 0, o, np.exp(o) - 1) for o in heads], axis=1)   # Hidden layer: ELU then concatenate\nfinal = np.mean(heads, axis=0)                                                          # Output layer: average directly\nprint(\"Three heads concatenated:\", hidden.shape, \" Three heads averaged:\", final.shape)\n```\n\n```text\nThree heads concatenated: (5, 6)  Three heads averaged: (5, 2)\n```\n\n[![Figure 1 of the paper: on the left, Wh_i and Wh_j are concatenated, dotted with a, passed through LeakyReLU, then softmaxed over j to obtain α_ij. On the right, node 1 applies three-head attention to six neighbors; three line styles represent the heads, whose outputs are concatenated or averaged into h'_1.](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)\n\n*Figure 1 from Veličković et al., taken from [**PDF page 4**](https://arxiv.org/pdf/1710.10903v3#page=4). The original caption is omitted.*\n\n### Cost and Properties\n\nA single head costs $O(|V|FF'+|E|F')$, the same order as GCN. Attention can be computed in parallel over edges, and outputs in parallel over nodes, without eigendecomposition. Multiple heads multiply storage and parameters by $K$, but run independently.\n\nThe attention mechanism is shared across edges and needs no advance knowledge of the entire graph. Directed graphs work by omitting $\\alpha_{ij}$ when $j\\rightarrow i$ is absent; unseen graphs can use the model directly. Compared with GraphSAGE, GAT examines the full neighborhood and assumes no neighbor order.\n\n## Discussion\n\n### Experimental Setup\n\nTransductive experiments use Cora, Citeseer and Pubmed with GCN's splits and 20 labels per class. The two-layer model has 8 first-layer heads of 8 dimensions each, 64 in total, followed by ELU; the second layer has one head outputting the classes. Regularization is heavy because training sets are small: L2 coefficient 0.0005 and dropout 0.6, applied even to attention coefficients. The authors note that attention dropout effectively gives each node a randomly sampled neighborhood every iteration. Pubmed has only 60 training nodes, so its output layer uses 8 heads and L2 increases to 0.001.\n\nInductive experiments use PPI, training on 20 graphs, validating on 2 and testing on 2 entirely unseen graphs. The three-layer model uses 4 heads of 256 dimensions in each of the first two layers; the final layer averages 6 heads of 121 dimensions and applies sigmoid for multilabel classification. With enough data, neither L2 nor dropout is used; intermediate layers add skip connections.\n\n### Main Results\n\n| Method | Cora | Citeseer | Pubmed |\n| --- | ---: | ---: | ---: |\n| GCN | 81.5% | 70.3% | 79.0% |\n| GCN-64* | 81.4 ± 0.5% | 70.9 ± 0.5% | 79.0 ± 0.3% |\n| **GAT** | **83.0 ± 0.7%** | **72.5 ± 0.7%** | **79.0 ± 0.3%** |\n\nGCN-64* widens GCN's hidden layer to GAT's 64 dimensions and reports the better result after trying ReLU and ELU, for a fairer assessment of attention itself. GAT exceeds GCN-64* by 1.6 percentage points on both Cora and Citeseer and ties on Pubmed.\n\nThe gap is much larger on PPI:\n\n| Method | PPI micro-F1 |\n| --- | ---: |\n| GraphSAGE-LSTM (original paper) | 0.612 |\n| GraphSAGE* (architecture tuned by the authors) | 0.768 |\n| Const-GAT | 0.934 ± 0.006 |\n| **GAT** | **0.973 ± 0.002** |\n\nRead this table in two steps. Const-GAT keeps the architecture but replaces attention with a constant, weighting all neighbors equally, close to GCN-style aggregation. The jump from 0.768 to 0.934 mostly comes from full neighborhoods and a larger architecture; the rise from 0.934 to 0.973 is attention's own 3.9-percentage-point contribution.\n\n### What Did It Learn?\n\nThe authors visualize a pretrained model's first-layer outputs on Cora with t-SNE. Seven classes form clear clusters in two dimensions, with edge widths showing attention averaged over eight heads. They caution that properly interpreting these coefficients requires more domain knowledge of the dataset, leaving that to future work.\n\n### The Authors' Own Limitations\n\n- Sparse GAT can make storage linear in nodes and edges, but the frameworks of the time support only sparse multiplication of two-dimensional tensors, limiting batches of multiple graphs.\n- Depending on graph regularity, GPUs may offer little advantage over CPUs for these sparse operations.\n- Receptive-field size is limited by depth, as in GCN.\n- Parallel computation over all edges repeats substantial work where neighborhoods overlap heavily.\n\n### Static Attention\n\nIn 2021, Brody, Alon and Yahav at Technion publish “How Attentive are Graph Attention Networks?”, identifying a structural limitation in GAT attention. [GATv2 abstract](https://arxiv.org/abs/2105.14491v3)\n\nLook back at the score: $a^\\top[Wh_i\\|Wh_j]$ splits into a term depending only on $i$ and one depending only on $j$. LeakyReLU and softmax are monotonic, so the **ranking** of candidate neighbors depends only on the $j$ term, independently of query node $i$. Make the example fully connected so everyone sees everyone:\n\n```python\nA = np.ones((5, 5))                      # Make fully connected so every node sees all others\nalpha, _ = gat_head(h, W, a)\nprint(alpha)\nprint(\"Argmax of each row:\", alpha.argmax(1))\n```\n\n```text\n[[0.37 0.19 0.17 0.15 0.13]\n [0.39 0.19 0.16 0.14 0.12]\n [0.4  0.2  0.17 0.12 0.11]\n [0.43 0.22 0.18 0.09 0.08]\n [0.44 0.22 0.18 0.09 0.07]]\nArgmax of each row: [0 0 0 0 0]\n```\n\nAll five nodes give exactly the same ranking: node 0 most important, node 4 least. Only the proportions differ. On the original graph, the mask limits each node to its own neighbors, creating apparent individual preferences. Whenever two candidate lists overlap, their rankings agree.\n\nBrody et al. call this **static attention**. Their fix changes the operation order: multiply $[h_i\\|h_j]$ by weights, apply LeakyReLU, then dot with $a$. Placing the nonlinearity in the middle lets rankings depend on the query node. This version, GATv2, beats GAT on 11 benchmarks including OGB with comparable parameter counts.\n\n\n> [!TIP]\n> \n> For GAT's predecessor GCN and GraphSAGE's approach to neighboring weights, see our notes:\n> \n> - **[[16.09] GCN: Is One Parameter Enough?](/blog/papers/graph-learning/gcn)**\n> - **[[17.06] GraphSAGE: Sampling the Roll Call](/blog/papers/graph-learning/graphsage)**\n\n\n## Conclusion\n\nGAT brings attention to graph neural networks, giving neighbors different, learnable, feature-dependent importance within the same neighborhood. Masking confines attention to neighbors and preserves graph structure; multiple heads stabilize training.\n\nOn citation networks it adds 1 to 2 percentage points over GCN. On PPI, a comparison within the same architecture clearly measures attention's contribution.\n\nRemember to apply dropout to attention coefficients: the paper relies on this for small datasets. If your task needs different nodes to prefer different neighbors, compare with GATv2.\n\nRemove the mask and let every node see every other node, and you have Transformer attention. Later Graph Transformers address precisely the graph structure lost at that step.\n","imageSizes":{"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png":{"width":1124,"height":540}}}