{"notes":[{"id":"graph-learning/gat","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GAT","subtitle":"邻居有轻重","author":"Z. Yuan","paperDate":"2017-10-30","publishedOn":"2023-11-10","world":"external/gat","paperTitle":"Graph Attention Networks","paperUrl":"https://arxiv.org/abs/1710.10903","bibliography":{"title":"Graph Attention Networks","authors":["Petar Veličković","Guillem Cucurull","Arantxa Casanova","Adriana Romero","Pietro Liò","Yoshua Bengio"],"url":"https://arxiv.org/abs/1710.10903","pdfUrl":"https://arxiv.org/pdf/1710.10903v3","codeUrl":"https://github.com/PetarV-/GAT","preprint":{"id":"1710.10903","submittedOn":"2017-10-30","url":"https://arxiv.org/abs/1710.10903"},"publication":{"venue":"ICLR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1710.10903"},"citation":"@misc{graphlearninggat2017,\n  title = {{Graph Attention Networks}},\n  author = {Petar Veličković and Guillem Cucurull and Arantxa Casanova and Adriana Romero and Pietro Liò and Yoshua Bengio},\n  year = {2017},\n  eprint = {1710.10903},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1710.10903}\n}\n","citationPath":"/content/papers/citations/graph-learning/gat.402fa57986a5.bib"},"thumbnail":{"src":"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png","width":1124,"height":540},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/gat.8d91c14ca422.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"解决问题","text":"解决问题","depth":2},{"id":"一个注意力头","text":"一个注意力头","depth":3},{"id":"多头","text":"多头","depth":3},{"id":"成本与特性","text":"成本与特性","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"学到了什么","text":"学到了什么","depth":3},{"id":"作者自己列的限制","text":"作者自己列的限制","depth":3},{"id":"静态的注意力","text":"静态的注意力","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gcn","relation":"referenced","topic":""},{"id":"graph-learning/graphsage","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一个参数够吗？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/gcn.0866efaf54eb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"把图当成正则项","text":"把图当成正则项","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"从频域一路简化","text":"从频域一路简化","depth":3},{"id":"一层-gcn","text":"一层 GCN","depth":3},{"id":"两层就够","text":"两层就够","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"实验设置","text":"实验设置","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"哪一种传播最好","text":"哪一种传播最好","depth":3},{"id":"叠深一点会怎样","text":"叠深一点会怎样","depth":3},{"id":"和-wl-算法的关系","text":"和 WL 算法的关系","depth":3},{"id":"四天后的一篇评论","text":"四天后的一篇评论","depth":3},{"id":"论文自己列的限制","text":"论文自己列的限制","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]},{"id":"graph-learning/graphsage","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphSAGE","subtitle":"抽样点名","author":"Z. Yuan","paperDate":"2017-06-07","publishedOn":"2023-10-21","world":"external/graphsage","paperTitle":"Inductive Representation Learning on Large Graphs","paperUrl":"https://arxiv.org/abs/1706.02216","bibliography":{"title":"Inductive Representation Learning on Large Graphs","authors":["William L. Hamilton","Rex Ying","Jure Leskovec"],"url":"https://arxiv.org/abs/1706.02216","pdfUrl":"https://arxiv.org/pdf/1706.02216v4","codeUrl":"https://github.com/williamleif/GraphSAGE","preprint":{"id":"1706.02216","submittedOn":"2017-06-07","url":"https://arxiv.org/abs/1706.02216"},"publication":{"venue":"NeurIPS","year":2017,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html"},"citation":"@misc{graphlearninggraphsage2017,\n  title = {{Inductive Representation Learning on Large Graphs}},\n  author = {William L. Hamilton and Rex Ying and Jure Leskovec},\n  year = {2017},\n  eprint = {1706.02216},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.02216}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphsage.1a0b3fd6e8b8.bib"},"thumbnail":{"src":"/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png","width":1090,"height":384},"locale":"zh-hans","path":"/content/papers/zh-hans/graph-learning/graphsage.e062006971fb.md","headings":[{"id":"定义问题","text":"定义问题","depth":2},{"id":"transductive-与-inductive","text":"transductive 与 inductive","depth":3},{"id":"邻居太多","text":"邻居太多","depth":3},{"id":"解决问题","text":"解决问题","depth":2},{"id":"算法","text":"算法","depth":3},{"id":"邻居用抽的","text":"邻居用抽的","depth":3},{"id":"三种聚合器","text":"三种聚合器","depth":3},{"id":"没有标签也能学","text":"没有标签也能学","depth":3},{"id":"和-wl-测试的关系","text":"和 WL 测试的关系","depth":3},{"id":"讨论","text":"讨论","depth":2},{"id":"三个数据集","text":"三个数据集","depth":3},{"id":"主要结果","text":"主要结果","depth":3},{"id":"速度与采样的取舍","text":"速度与采样的取舍","depth":3},{"id":"后来的事","text":"后来的事","depth":3},{"id":"结论","text":"结论","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gat","relation":"citing","topic":""}]}],"text":"注意力机制在机器翻译上证明了自己之后，2017 年 6 月，Transformer 干脆只用注意力，把 RNN 整个拿掉。\n\n注意力做的事很单纯。每个位置看看其他位置，依相似程度决定各拿多少，再加权汇整。这和图神经网络的聚合几乎是同一件事，只差在权重从哪里来。\n\nGCN 的权重来自度数，节点 $i$ 收到邻居 $j$ 的比例固定是 $1/\\sqrt{\\tilde d_i\\tilde d_j}$。剑桥大学的 Petar Veličković 和 Bengio 等人想让这个比例由特征决定，于是有了 GAT。\n\n## 定义问题\n\n作者整理了当时两条路线各自的麻烦。\n\n频域方法以拉普拉斯矩阵的特征分解为基础，学出来的滤波器绑在那张图的特征向量上。换一张结构不同的图，模型就不能直接用。GCN 把滤波器简化成只看一步的邻居，算起来快了，但同一个邻域里，每个邻居分到的份量只由度数决定。\n\n空间方法要处理邻居数量不一的问题。GraphSAGE 每个节点固定抽样几个邻居，推理时看不到完整的邻域；它表现最好的 LSTM 聚合器还得假设邻居有顺序，只好每次把邻居随机打乱再喂进去。\n\n作者想要的层有几个条件。能并行计算；能处理度数不同的节点；同一个邻域里的邻居可以有不同的份量；而且不需要事先知道整张图的结构，可以直接用在没见过的图上。\n\n## 解决问题\n\n### 一个注意力头\n\n输入是每个节点的特征 $h_i\\in\\mathbb{R}^F$。第一步，所有节点共用一个线性变换 $W\\in\\mathbb{R}^{F'\\times F}$。接着对每一对节点算一个分数：\n\n$$\ne_{ij}=a\\left(Wh_i,\\ Wh_j\\right)\n$$\n\n$e_{ij}$ 代表节点 $j$ 的特征对节点 $i$ 有多重要。最一般的形式下，每个节点可以看所有节点，但那样就把图的结构整个丢掉了。作者的做法是**遮罩注意力**，只对 $i$ 的邻居 $\\mathcal N_i$（包含自己）计算，再用 softmax 归一化：\n\n$$\n\\alpha_{ij}=\\mathrm{softmax}_j(e_{ij})=\\frac{\\exp(e_{ij})}{\\sum_{k\\in\\mathcal N_i}\\exp(e_{ik})}\n$$\n\n注意力机制 $a$ 是一层前馈网络，权重向量 $a\\in\\mathbb{R}^{2F'}$，接 LeakyReLU（负斜率 0.2）。展开来是：\n\n$$\n\\alpha_{ij}=\\frac{\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_j\\right]\\right)\\right)}{\\sum_{k\\in\\mathcal N_i}\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_k\\right]\\right)\\right)}\n$$\n\n$\\|$ 是串接。最后用这些系数把邻居加权汇整，$h'_i=\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha_{ij}Wh_j\\right)$。\n\n在一张五个节点的小图上跑一个头。这里把 $a^\\top[Wh_i\\|Wh_j]$ 拆成两半，前半段只看 $i$、后半段只看 $j$，就能一次算出整个分数矩阵：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(1)\n\n# 5 个节点：0-1、0-2、1-2、2-3、3-4，加上自环\nA = np.eye(5)\nfor i, j in [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]:\n    A[i, j] = A[j, i] = 1\n\nh = rng.normal(size=(5, 3))          # 每个节点 3 维特征\nW = rng.normal(size=(3, 2))          # 共用的线性变换，输出 2 维\na = rng.normal(size=4)               # 注意力向量，长度 2F' = 4\n\ndef leaky_relu(z, slope=0.2):\n    return np.where(z > 0, z, slope * z)\n\ndef gat_head(h, W, a):\n    Wh = h @ W                                             # (5, 2)\n    e = leaky_relu((Wh @ a[:2])[:, None] + (Wh @ a[2:])[None, :])   # a^T [Wh_i || Wh_j]，(5, 5)\n    e = np.where(A > 0, e, -np.inf)                        # 遮罩：只留邻居\n    alpha = np.exp(e - e.max(1, keepdims=True))\n    alpha /= alpha.sum(1, keepdims=True)                   # 每行在邻居之间做 softmax\n    return alpha, alpha @ Wh\n\nalpha, out = gat_head(h, W, a)\nprint(\"注意力系数:\")\nprint(alpha)\nprint(\"输出:\", out.shape)\n```\n\n```text\n注意力系数:\n[[0.52 0.26 0.23 0.   0.  ]\n [0.52 0.26 0.22 0.   0.  ]\n [0.45 0.23 0.19 0.14 0.  ]\n [0.   0.   0.51 0.26 0.23]\n [0.   0.   0.   0.58 0.42]]\n输出: (5, 2)\n```\n\n每一行加起来是 1，不是邻居的位置都是 0。同样有三个邻居的 0 号和 1 号，GCN 会给每个邻居差不多的份量；这里的份量则由特征决定，0 号节点拿到的最多。\n\n### 多头\n\n和 Transformer 一样，作者发现多头注意力能让训练更稳定。$K$ 个头各自有自己的 $W^k$ 和 $a^k$，独立算完之后，中间层把结果串接起来，输出维度变成 $KF'$；最后一层如果还串接就没有意义了，改成取平均，再接 softmax 或 sigmoid：\n\n$$\nh'_i=\\Big\\|_{k=1}^{K}\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right),\\qquad\nh'_i=\\sigma\\left(\\frac1K\\sum_{k=1}^K\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right)\n$$\n\n接着上面的例子，换三组权重，看两种合并方式的形状：\n\n```python\nheads = [gat_head(h, rng.normal(size=(3, 2)), rng.normal(size=4))[1] for _ in range(3)]\nhidden = np.concatenate([np.where(o > 0, o, np.exp(o) - 1) for o in heads], axis=1)   # 中间层：ELU 后串接\nfinal = np.mean(heads, axis=0)                                                          # 输出层：直接平均\nprint(\"三头串接:\", hidden.shape, \" 三头平均:\", final.shape)\n```\n\n```text\n三头串接: (5, 6)  三头平均: (5, 2)\n```\n\n[![原论文 Figure 1：左图是注意力机制，Wh_i 与 Wh_j 串接后与向量 a 做内积，经过 LeakyReLU，再对 j 做 softmax 得到 α_ij；右图是节点 1 对六个邻居做三头注意力，三种线条代表三个头，结果串接或平均后得到 h'_1。](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)\n\n*原论文 Figure 1（Veličković 等人），取自 [**PDF 第 4 页**](https://arxiv.org/pdf/1710.10903v3#page=4)。图注未收录。*\n\n### 成本与特性\n\n单一个头的计算量是 $O(|V|FF'+|E|F')$，和 GCN 同一个量级；注意力可以沿着所有边并行计算，输出可以沿着所有节点并行计算，不需要特征分解。多头的存储量与参数量会乘上 $K$，但每个头彼此独立。\n\n注意力在每条边上共用，不需要事先知道整张图。所以有向图也能处理，边 $j\\rightarrow i$ 不存在就不算 $\\alpha_{ij}$；没见过的新图也能直接应用。和 GraphSAGE 相比，它看的是完整的邻域，也不假设邻居有顺序。\n\n## 讨论\n\n### 实验设置\n\ntransductive 的实验是 Cora、Citeseer、Pubmed 三个引用网络，沿用 GCN 的切分，每类 20 个标签。模型两层，第一层 8 个头、每头 8 维，共 64 维，接 ELU；第二层一个头直接输出类别数。训练集很小，所以正则化下得很重。L2 系数 0.0005，dropout 0.6，连注意力系数也做 dropout；作者特别指出，这等于每次迭代都让节点看到一个随机采样的邻域。Pubmed 只有 60 个训练节点，输出层改用 8 个头，L2 也加重到 0.001。\n\ninductive 的实验是 PPI，20 张图训练、2 张验证、2 张测试，测试的图完全没见过。模型三层，前两层各 4 个头、每头 256 维，最后一层 6 个头、每头 121 维取平均，接 sigmoid 做多标签分类。这里数据够多，不用 L2 和 dropout，中间层加了 skip connection。\n\n### 主要结果\n\n| 方法 | Cora | Citeseer | Pubmed |\n| --- | ---: | ---: | ---: |\n| GCN | 81.5% | 70.3% | 79.0% |\n| GCN-64* | 81.4 ± 0.5% | 70.9 ± 0.5% | 79.0 ± 0.3% |\n| **GAT** | **83.0 ± 0.7%** | **72.5 ± 0.7%** | **79.0 ± 0.3%** |\n\nGCN-64* 是作者把 GCN 的隐藏层加到和 GAT 一样的 64 维、试过 ReLU 和 ELU 之后的最好结果，目的是公平评估注意力本身的贡献。GAT 在 Cora、Citeseer 都比 GCN-64* 高 1.6 个百分点，Pubmed 打平。\n\nPPI 上的差距大得多：\n\n| 方法 | PPI micro-F1 |\n| --- | ---: |\n| GraphSAGE-LSTM（原论文） | 0.612 |\n| GraphSAGE*（作者调过架构） | 0.768 |\n| Const-GAT | 0.934 ± 0.006 |\n| **GAT** | **0.973 ± 0.002** |\n\n读这张表要分两段。Const-GAT 是同一个架构、注意力换成常数，每个邻居一样重，接近 GCN 式的聚合。从 0.768 到 0.934，主要来自看完整的邻域与更大的架构；从 0.934 到 0.973，才是注意力本身多出来的 3.9 个百分点。\n\n### 学到了什么\n\n作者把 Cora 上预训练模型的第一层输出做 t-SNE，七个类别在二维平面上分成了清楚的群，边的粗细是八个头平均的注意力。他们也提醒，要正确解读这些注意力系数，需要对数据集本身有更多的领域知识，这部分留给未来的工作。\n\n### 作者自己列的限制\n\n- 稀疏矩阵版本的 GAT 能把存储成本压到和节点、边数成线性，但当时的框架只支持二维张量的稀疏乘法，批量处理多张图时很受限。\n- 视图的规则程度，在这种稀疏运算上，GPU 未必比 CPU 快多少。\n- 感受野的大小受网络深度限制，这点和 GCN 一样。\n- 沿着所有边并行计算，在邻域高度重叠的图上会有很多重复运算。\n\n### 静态的注意力\n\n2021 年，Technion 的 Brody、Alon 与 Yahav 发表〈How Attentive are Graph Attention Networks?〉，指出 GAT 的注意力有一个结构上的限制。[GATv2 摘要](https://arxiv.org/abs/2105.14491v3)\n\n回头看分数的算法，$a^\\top[Wh_i\\|Wh_j]$ 可以拆成「只和 $i$ 有关的一项」加上「只和 $j$ 有关的一项」。LeakyReLU 和 softmax 都是单调的，所以每个节点对候选邻居的**排序**，只由 $j$ 那一项决定，和查询节点 $i$ 无关。把刚才的小图改成全连接，让每个节点都能看所有人：\n\n```python\nA = np.ones((5, 5))                      # 改成全连接，每个节点都能看所有人\nalpha, _ = gat_head(h, W, a)\nprint(alpha)\nprint(\"每一行最大的是:\", alpha.argmax(1))\n```\n\n```text\n[[0.37 0.19 0.17 0.15 0.13]\n [0.39 0.19 0.16 0.14 0.12]\n [0.4  0.2  0.17 0.12 0.11]\n [0.43 0.22 0.18 0.09 0.08]\n [0.44 0.22 0.18 0.09 0.07]]\n每一行最大的是: [0 0 0 0 0]\n```\n\n五个节点的排序完全一样，都是 0 号最重要，4 号最不重要；不同的只有分配的比例。在原本的图上，遮罩让每个节点只能从自己的邻居里挑，看起来各有偏好，但只要两个节点的候选名单重叠，它们的排序就会一致。\n\nBrody 等人把这称为**静态注意力**。他们的修正是调换运算顺序，先把 $[h_i\\|h_j]$ 乘上权重、过 LeakyReLU，最后才和 $a$ 做内积，让非线性夹在中间，排序就能随查询节点改变。这个版本叫 GATv2，在 11 个 OGB 等基准上都胜过 GAT，参数量相当。\n\n\n> [!TIP]\n> \n> GAT 的前身 GCN，以及同样处理邻居权重的 GraphSAGE，可以参考我们的笔记：\n> \n> - **[[16.09] GCN：一个参数够吗？](/blog/papers/graph-learning/gcn)**\n> - **[[17.06] GraphSAGE：抽样点名](/blog/papers/graph-learning/graphsage)**\n\n\n## 结论\n\nGAT 把注意力搬进图神经网络，让同一个邻域里的邻居可以有不同的份量，而且份量由特征决定、可以学。遮罩把注意力限制在邻居之间，图的结构就保留下来；多头让训练更稳定。\n\n在引用网络上，它比 GCN 多了 1 到 2 个百分点；在 PPI 上，同架构的对照把注意力本身的贡献量得很清楚。\n\n用的时候记得把注意力系数也做 dropout，论文在小数据集上靠的就是这一招。如果你的任务需要「不同节点偏好不同邻居」，就换成 GATv2 比比看。\n\n把遮罩拿掉，让每个节点看所有节点，就是 Transformer 的注意力。后来的 Graph Transformer，处理的正是这一步丢掉的图结构。\n","imageSizes":{"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png":{"width":1124,"height":540}}}