{"notes":[{"id":"graph-learning/gat","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GAT","subtitle":"鄰居有輕重","author":"Z. Yuan","paperDate":"2017-10-30","publishedOn":"2023-11-10","world":"external/gat","paperTitle":"Graph Attention Networks","paperUrl":"https://arxiv.org/abs/1710.10903","bibliography":{"title":"Graph Attention Networks","authors":["Petar Veličković","Guillem Cucurull","Arantxa Casanova","Adriana Romero","Pietro Liò","Yoshua Bengio"],"url":"https://arxiv.org/abs/1710.10903","pdfUrl":"https://arxiv.org/pdf/1710.10903v3","codeUrl":"https://github.com/PetarV-/GAT","preprint":{"id":"1710.10903","submittedOn":"2017-10-30","url":"https://arxiv.org/abs/1710.10903"},"publication":{"venue":"ICLR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1710.10903"},"citation":"@misc{graphlearninggat2017,\n  title = {{Graph Attention Networks}},\n  author = {Petar Veličković and Guillem Cucurull and Arantxa Casanova and Adriana Romero and Pietro Liò and Yoshua Bengio},\n  year = {2017},\n  eprint = {1710.10903},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1710.10903}\n}\n","citationPath":"/content/papers/citations/graph-learning/gat.402fa57986a5.bib"},"thumbnail":{"src":"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png","width":1124,"height":540},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/gat.99de7ab34688.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"一個注意力頭","text":"一個注意力頭","depth":3},{"id":"多頭","text":"多頭","depth":3},{"id":"成本與特性","text":"成本與特性","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"學到了什麼","text":"學到了什麼","depth":3},{"id":"作者自己列的限制","text":"作者自己列的限制","depth":3},{"id":"靜態的注意力","text":"靜態的注意力","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gcn","relation":"referenced","topic":""},{"id":"graph-learning/graphsage","relation":"referenced","topic":""}]},{"id":"graph-learning/gcn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GCN","subtitle":"一個參數夠嗎？","author":"Z. Yuan","paperDate":"2016-09-09","publishedOn":"2023-10-01","world":"external/gcn","paperTitle":"Semi-Supervised Classification with Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1609.02907","bibliography":{"title":"Semi-Supervised Classification with Graph Convolutional Networks","authors":["Thomas N. Kipf","Max Welling"],"url":"https://arxiv.org/abs/1609.02907","pdfUrl":"https://arxiv.org/pdf/1609.02907v4","codeUrl":"https://github.com/tkipf/gcn","preprint":{"id":"1609.02907","submittedOn":"2016-09-09","url":"https://arxiv.org/abs/1609.02907"},"publication":{"venue":"ICLR","year":2017,"kind":"conference","url":"https://arxiv.org/abs/1609.02907"},"citation":"@misc{graphlearninggcn2016,\n  title = {{Semi-Supervised Classification with Graph Convolutional Networks}},\n  author = {Thomas N. Kipf and Max Welling},\n  year = {2016},\n  eprint = {1609.02907},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1609.02907}\n}\n","citationPath":"/content/papers/citations/graph-learning/gcn.43058ec3e77e.bib"},"thumbnail":{"src":"/content/papers/assets/ded600c9ca8f4eda09d39ffb16b3c9b0ec730780063e55948b29924d00f14794.png","width":1122,"height":400},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/gcn.66a3d9542810.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"把圖當成正則項","text":"把圖當成正則項","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"從頻域一路簡化","text":"從頻域一路簡化","depth":3},{"id":"一層-gcn","text":"一層 GCN","depth":3},{"id":"兩層就夠","text":"兩層就夠","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"哪一種傳播最好","text":"哪一種傳播最好","depth":3},{"id":"疊深一點會怎樣","text":"疊深一點會怎樣","depth":3},{"id":"和-wl-演算法的關係","text":"和 WL 演算法的關係","depth":3},{"id":"四天後的一篇評論","text":"四天後的一篇評論","depth":3},{"id":"論文自己列的限制","text":"論文自己列的限制","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/spectral-networks","relation":"referenced","topic":""},{"id":"graph-learning/chebnet","relation":"referenced","topic":""}]},{"id":"graph-learning/graphsage","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphSAGE","subtitle":"抽樣點名","author":"Z. Yuan","paperDate":"2017-06-07","publishedOn":"2023-10-21","world":"external/graphsage","paperTitle":"Inductive Representation Learning on Large Graphs","paperUrl":"https://arxiv.org/abs/1706.02216","bibliography":{"title":"Inductive Representation Learning on Large Graphs","authors":["William L. Hamilton","Rex Ying","Jure Leskovec"],"url":"https://arxiv.org/abs/1706.02216","pdfUrl":"https://arxiv.org/pdf/1706.02216v4","codeUrl":"https://github.com/williamleif/GraphSAGE","preprint":{"id":"1706.02216","submittedOn":"2017-06-07","url":"https://arxiv.org/abs/1706.02216"},"publication":{"venue":"NeurIPS","year":2017,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html"},"citation":"@misc{graphlearninggraphsage2017,\n  title = {{Inductive Representation Learning on Large Graphs}},\n  author = {William L. Hamilton and Rex Ying and Jure Leskovec},\n  year = {2017},\n  eprint = {1706.02216},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.02216}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphsage.1a0b3fd6e8b8.bib"},"thumbnail":{"src":"/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png","width":1090,"height":384},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphsage.fecc6d2df60f.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"transductive-與-inductive","text":"transductive 與 inductive","depth":3},{"id":"鄰居太多","text":"鄰居太多","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"演算法","text":"演算法","depth":3},{"id":"鄰居用抽的","text":"鄰居用抽的","depth":3},{"id":"三種聚合器","text":"三種聚合器","depth":3},{"id":"沒有標籤也能學","text":"沒有標籤也能學","depth":3},{"id":"和-wl-測試的關係","text":"和 WL 測試的關係","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"三個資料集","text":"三個資料集","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"速度與取樣的取捨","text":"速度與取樣的取捨","depth":3},{"id":"後來的事","text":"後來的事","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gat","relation":"citing","topic":""}]}],"text":"注意力機制在機器翻譯上證明了自己之後，2017 年 6 月，Transformer 乾脆只用注意力，把 RNN 整個拿掉。\n\n注意力做的事很單純。每個位置看看其他位置，依相似程度決定各拿多少，再加權彙整。這和圖神經網路的聚合幾乎是同一件事，只差在權重從哪裡來。\n\nGCN 的權重來自度數，節點 $i$ 收到鄰居 $j$ 的比例固定是 $1/\\sqrt{\\tilde d_i\\tilde d_j}$。劍橋大學的 Petar Veličković 和 Bengio 等人想讓這個比例由特徵決定，於是有了 GAT。\n\n## 定義問題\n\n作者整理了當時兩條路線各自的麻煩。\n\n頻域方法以拉普拉斯矩陣的特徵分解為基礎，學出來的濾波器綁在那張圖的特徵向量上。換一張結構不同的圖，模型就不能直接用。GCN 把濾波器簡化成只看一步的鄰居，算起來快了，但同一個鄰域裡，每個鄰居分到的份量只由度數決定。\n\n空間方法要處理鄰居數量不一的問題。GraphSAGE 每個節點固定抽樣幾個鄰居，推論時看不到完整的鄰域；它表現最好的 LSTM 聚合器還得假設鄰居有順序，只好每次把鄰居隨機打亂再餵進去。\n\n作者想要的層有幾個條件。能平行計算；能處理度數不同的節點；同一個鄰域裡的鄰居可以有不同的份量；而且不需要事先知道整張圖的結構，可以直接用在沒見過的圖上。\n\n## 解決問題\n\n### 一個注意力頭\n\n輸入是每個節點的特徵 $h_i\\in\\mathbb{R}^F$。第一步，所有節點共用一個線性轉換 $W\\in\\mathbb{R}^{F'\\times F}$。接著對每一對節點算一個分數：\n\n$$\ne_{ij}=a\\left(Wh_i,\\ Wh_j\\right)\n$$\n\n$e_{ij}$ 代表節點 $j$ 的特徵對節點 $i$ 有多重要。最一般的形式下，每個節點可以看所有節點，但那樣就把圖的結構整個丟掉了。作者的做法是**遮罩注意力**，只對 $i$ 的鄰居 $\\mathcal N_i$（包含自己）計算，再用 softmax 正規化：\n\n$$\n\\alpha_{ij}=\\mathrm{softmax}_j(e_{ij})=\\frac{\\exp(e_{ij})}{\\sum_{k\\in\\mathcal N_i}\\exp(e_{ik})}\n$$\n\n注意力機制 $a$ 是一層前饋網路，權重向量 $a\\in\\mathbb{R}^{2F'}$，接 LeakyReLU（負斜率 0.2）。展開來是：\n\n$$\n\\alpha_{ij}=\\frac{\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_j\\right]\\right)\\right)}{\\sum_{k\\in\\mathcal N_i}\\exp\\left(\\mathrm{LeakyReLU}\\left(a^\\top\\left[Wh_i\\,\\|\\,Wh_k\\right]\\right)\\right)}\n$$\n\n$\\|$ 是串接。最後用這些係數把鄰居加權彙整，$h'_i=\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha_{ij}Wh_j\\right)$。\n\n在一張五個節點的小圖上跑一個頭。這裡把 $a^\\top[Wh_i\\|Wh_j]$ 拆成兩半，前半段只看 $i$、後半段只看 $j$，就能一次算出整個分數矩陣：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(1)\n\n# 5 個節點：0-1、0-2、1-2、2-3、3-4，加上自環\nA = np.eye(5)\nfor i, j in [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]:\n    A[i, j] = A[j, i] = 1\n\nh = rng.normal(size=(5, 3))          # 每個節點 3 維特徵\nW = rng.normal(size=(3, 2))          # 共用的線性轉換，輸出 2 維\na = rng.normal(size=4)               # 注意力向量，長度 2F' = 4\n\ndef leaky_relu(z, slope=0.2):\n    return np.where(z > 0, z, slope * z)\n\ndef gat_head(h, W, a):\n    Wh = h @ W                                             # (5, 2)\n    e = leaky_relu((Wh @ a[:2])[:, None] + (Wh @ a[2:])[None, :])   # a^T [Wh_i || Wh_j]，(5, 5)\n    e = np.where(A > 0, e, -np.inf)                        # 遮罩：只留鄰居\n    alpha = np.exp(e - e.max(1, keepdims=True))\n    alpha /= alpha.sum(1, keepdims=True)                   # 每列在鄰居之間做 softmax\n    return alpha, alpha @ Wh\n\nalpha, out = gat_head(h, W, a)\nprint(\"注意力係數:\")\nprint(alpha)\nprint(\"輸出:\", out.shape)\n```\n\n```text\n注意力係數:\n[[0.52 0.26 0.23 0.   0.  ]\n [0.52 0.26 0.22 0.   0.  ]\n [0.45 0.23 0.19 0.14 0.  ]\n [0.   0.   0.51 0.26 0.23]\n [0.   0.   0.   0.58 0.42]]\n輸出: (5, 2)\n```\n\n每一列加起來是 1，不是鄰居的位置都是 0。同樣有三個鄰居的 0 號和 1 號，GCN 會給每個鄰居差不多的份量；這裡的份量則由特徵決定，0 號節點拿到的最多。\n\n### 多頭\n\n和 Transformer 一樣，作者發現多頭注意力能讓訓練更穩定。$K$ 個頭各自有自己的 $W^k$ 和 $a^k$，獨立算完之後，中間層把結果串接起來，輸出維度變成 $KF'$；最後一層如果還串接就沒有意義了，改成取平均，再接 softmax 或 sigmoid：\n\n$$\nh'_i=\\Big\\|_{k=1}^{K}\\sigma\\left(\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right),\\qquad\nh'_i=\\sigma\\left(\\frac1K\\sum_{k=1}^K\\sum_{j\\in\\mathcal N_i}\\alpha^k_{ij}W^kh_j\\right)\n$$\n\n接著上面的例子，換三組權重，看兩種合併方式的形狀：\n\n```python\nheads = [gat_head(h, rng.normal(size=(3, 2)), rng.normal(size=4))[1] for _ in range(3)]\nhidden = np.concatenate([np.where(o > 0, o, np.exp(o) - 1) for o in heads], axis=1)   # 中間層：ELU 後串接\nfinal = np.mean(heads, axis=0)                                                          # 輸出層：直接平均\nprint(\"三頭串接:\", hidden.shape, \" 三頭平均:\", final.shape)\n```\n\n```text\n三頭串接: (5, 6)  三頭平均: (5, 2)\n```\n\n[![原論文 Figure 1：左圖是注意力機制，Wh_i 與 Wh_j 串接後與向量 a 做內積，經過 LeakyReLU，再對 j 做 softmax 得到 α_ij；右圖是節點 1 對六個鄰居做三頭注意力，三種線條代表三個頭，結果串接或平均後得到 h'_1。](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)](/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png)\n\n*原論文 Figure 1（Veličković 等人），取自 [**PDF 第 4 頁**](https://arxiv.org/pdf/1710.10903v3#page=4)。圖說未收錄。*\n\n### 成本與特性\n\n單一個頭的計算量是 $O(|V|FF'+|E|F')$，和 GCN 同一個量級；注意力可以沿著所有邊平行計算，輸出可以沿著所有節點平行計算，不需要特徵分解。多頭的儲存與參數量會乘上 $K$，但每個頭彼此獨立。\n\n注意力在每條邊上共用，不需要事先知道整張圖。所以有向圖也能處理，邊 $j\\rightarrow i$ 不存在就不算 $\\alpha_{ij}$；沒見過的新圖也能直接套用。和 GraphSAGE 相比，它看的是完整的鄰域，也不假設鄰居有順序。\n\n## 討論\n\n### 實驗設定\n\ntransductive 的實驗是 Cora、Citeseer、Pubmed 三個引用網路，沿用 GCN 的切分，每類 20 個標籤。模型兩層，第一層 8 個頭、每頭 8 維，共 64 維，接 ELU；第二層一個頭直接輸出類別數。訓練集很小，所以正則化下得很重。L2 係數 0.0005，dropout 0.6，連注意力係數也做 dropout；作者特別指出，這等於每次迭代都讓節點看到一個隨機取樣的鄰域。Pubmed 只有 60 個訓練節點，輸出層改用 8 個頭，L2 也加重到 0.001。\n\ninductive 的實驗是 PPI，20 張圖訓練、2 張驗證、2 張測試，測試的圖完全沒見過。模型三層，前兩層各 4 個頭、每頭 256 維，最後一層 6 個頭、每頭 121 維取平均，接 sigmoid 做多標籤分類。這裡資料夠多，不用 L2 和 dropout，中間層加了 skip connection。\n\n### 主要結果\n\n| 方法 | Cora | Citeseer | Pubmed |\n| --- | ---: | ---: | ---: |\n| GCN | 81.5% | 70.3% | 79.0% |\n| GCN-64* | 81.4 ± 0.5% | 70.9 ± 0.5% | 79.0 ± 0.3% |\n| **GAT** | **83.0 ± 0.7%** | **72.5 ± 0.7%** | **79.0 ± 0.3%** |\n\nGCN-64* 是作者把 GCN 的隱藏層加到和 GAT 一樣的 64 維、試過 ReLU 和 ELU 之後的最好結果，目的是公平評估注意力本身的貢獻。GAT 在 Cora、Citeseer 都比 GCN-64* 高 1.6 個百分點，Pubmed 打平。\n\nPPI 上的差距大得多：\n\n| 方法 | PPI micro-F1 |\n| --- | ---: |\n| GraphSAGE-LSTM（原論文） | 0.612 |\n| GraphSAGE*（作者調過架構） | 0.768 |\n| Const-GAT | 0.934 ± 0.006 |\n| **GAT** | **0.973 ± 0.002** |\n\n讀這張表要分兩段。Const-GAT 是同一個架構、注意力換成常數，每個鄰居一樣重，接近 GCN 式的聚合。從 0.768 到 0.934，主要來自看完整的鄰域與更大的架構；從 0.934 到 0.973，才是注意力本身多出來的 3.9 個百分點。\n\n### 學到了什麼\n\n作者把 Cora 上預訓練模型的第一層輸出做 t-SNE，七個類別在二維平面上分成了清楚的群，邊的粗細是八個頭平均的注意力。他們也提醒，要正確解讀這些注意力係數，需要對資料集本身有更多的領域知識，這部分留給未來的工作。\n\n### 作者自己列的限制\n\n- 稀疏矩陣版本的 GAT 能把儲存成本壓到和節點、邊數成線性，但當時的框架只支援二維張量的稀疏乘法，批次處理多張圖時很受限。\n- 視圖的規則程度，在這種稀疏運算上，GPU 未必比 CPU 快多少。\n- 感受野的大小受網路深度限制，這點和 GCN 一樣。\n- 沿著所有邊平行計算，在鄰域高度重疊的圖上會有很多重複運算。\n\n### 靜態的注意力\n\n2021 年，Technion 的 Brody、Alon 與 Yahav 發表〈How Attentive are Graph Attention Networks?〉，指出 GAT 的注意力有一個結構上的限制。[GATv2 摘要](https://arxiv.org/abs/2105.14491v3)\n\n回頭看分數的算法，$a^\\top[Wh_i\\|Wh_j]$ 可以拆成「只和 $i$ 有關的一項」加上「只和 $j$ 有關的一項」。LeakyReLU 和 softmax 都是單調的，所以每個節點對候選鄰居的**排序**，只由 $j$ 那一項決定，和查詢的節點 $i$ 無關。把剛才的小圖改成全連接，讓每個節點都能看所有人：\n\n```python\nA = np.ones((5, 5))                      # 改成全連接，每個節點都能看所有人\nalpha, _ = gat_head(h, W, a)\nprint(alpha)\nprint(\"每一列最大的是:\", alpha.argmax(1))\n```\n\n```text\n[[0.37 0.19 0.17 0.15 0.13]\n [0.39 0.19 0.16 0.14 0.12]\n [0.4  0.2  0.17 0.12 0.11]\n [0.43 0.22 0.18 0.09 0.08]\n [0.44 0.22 0.18 0.09 0.07]]\n每一列最大的是: [0 0 0 0 0]\n```\n\n五個節點的排序完全一樣，都是 0 號最重要，4 號最不重要；不同的只有分配的比例。在原本的圖上，遮罩讓每個節點只能從自己的鄰居裡挑，看起來各有偏好，但只要兩個節點的候選名單重疊，它們的排序就會一致。\n\nBrody 等人把這稱為**靜態注意力**。他們的修正是調換運算順序，先把 $[h_i\\|h_j]$ 乘上權重、過 LeakyReLU，最後才和 $a$ 做內積，讓非線性夾在中間，排序就能隨查詢節點改變。這個版本叫 GATv2，在 11 個 OGB 等基準上都勝過 GAT，參數量相當。\n\n\n> [!TIP]\n> \n> GAT 的前身 GCN，以及同樣處理鄰居權重的 GraphSAGE，可以參考我們的筆記：\n> \n> - **[[16.09] GCN：一個參數夠嗎？](/blog/papers/graph-learning/gcn)**\n> - **[[17.06] GraphSAGE：抽樣點名](/blog/papers/graph-learning/graphsage)**\n\n\n## 結論\n\nGAT 把注意力搬進圖神經網路，讓同一個鄰域裡的鄰居可以有不同的份量，而且份量由特徵決定、可以學。遮罩把注意力限制在鄰居之間，圖的結構就保留下來；多頭讓訓練更穩定。\n\n在引用網路上，它比 GCN 多了 1 到 2 個百分點；在 PPI 上，同架構的對照把注意力本身的貢獻量得很清楚。\n\n用的時候記得把注意力係數也做 dropout，論文在小資料集上靠的就是這一招。如果你的任務需要「不同節點偏好不同鄰居」，就換成 GATv2 比比看。\n\n把遮罩拿掉，讓每個節點看所有節點，就是 Transformer 的注意力。後來的 Graph Transformer，處理的正是這一步丟掉的圖結構。\n","imageSizes":{"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png":{"width":1124,"height":540}}}