{"notes":[{"id":"graph-learning/graphormer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graphormer","subtitle":"其實不差","author":"Z. Yuan","paperDate":"2021-06-09","publishedOn":"2024-03-28","world":"external/graphormer","paperTitle":"Do Transformers Really Perform Bad for Graph Representation?","paperUrl":"https://arxiv.org/abs/2106.05234","bibliography":{"title":"Do Transformers Really Perform Bad for Graph Representation?","authors":["Chengxuan Ying","Tianle Cai","Shengjie Luo","Shuxin Zheng","Guolin Ke","Di He","Yanming Shen","Tie-Yan Liu"],"url":"https://arxiv.org/abs/2106.05234","pdfUrl":"https://arxiv.org/pdf/2106.05234v5","codeUrl":"https://github.com/Microsoft/Graphormer","preprint":{"id":"2106.05234","submittedOn":"2021-06-09","url":"https://arxiv.org/abs/2106.05234"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html"},"citation":"@misc{graphlearninggraphormer2021,\n  title = {{Do Transformers Really Perform Bad for Graph Representation?}},\n  author = {Chengxuan Ying and Tianle Cai and Shengjie Luo and Shuxin Zheng and Guolin Ke and Di He and Yanming Shen and Tie-Yan Liu},\n  year = {2021},\n  eprint = {2106.05234},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.05234}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphormer.23b94dbe8ef3.bib"},"thumbnail":{"src":"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png","width":872,"height":620},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphormer.848ae0c7733a.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"中心性編碼","text":"中心性編碼","depth":3},{"id":"空間編碼","text":"空間編碼","depth":3},{"id":"邊編碼","text":"邊編碼","depth":3},{"id":"其他細節","text":"其他細節","depth":3},{"id":"能表示哪些-gnn","text":"能表示哪些 GNN？","depth":3},{"id":"比-wl-更強","text":"比 WL 更強","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"pcqm4m-lsc","text":"PCQM4M-LSC","depth":3},{"id":"其他資料集","text":"其他資料集","depth":3},{"id":"每個編碼貢獻多少","text":"每個編碼貢獻多少","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""}]},{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"連滿反而糟","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graph-transformer.1a6743ef39d8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"只在鄰居之間算注意力","text":"只在鄰居之間算注意力","depth":3},{"id":"拉普拉斯位置編碼","text":"拉普拉斯位置編碼","depth":3},{"id":"邊特徵與正規化","text":"邊特徵與正規化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"全連接會怎樣","text":"全連接會怎樣？","depth":3},{"id":"位置編碼與-batchnorm","text":"位置編碼與 BatchNorm","depth":3},{"id":"和-gnn-比","text":"和 GNN 比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"聽見圖的形狀","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/san.e84a43ca7dce.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"學出來的位置編碼","text":"學出來的位置編碼","depth":3},{"id":"全連接但分開兩套注意力","text":"全連接，但分開兩套注意力","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"gamma-該設多大","text":"\\gamma 該設多大？","depth":3},{"id":"全連接和-lpe-各自的貢獻","text":"全連接和 LPE 各自的貢獻","depth":3},{"id":"和當時最好的模型比","text":"和當時最好的模型比","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]}],"text":"2021 年，Transformer 已經是自然語言處理的主流，在電腦視覺也站穩了腳步。圖的排行榜卻是另一回事。\n\nOGB 大規模挑戰賽的 PCQM4M-LSC 有 380 多萬個分子，要從分子圖預測一個量子化學性質。這個資料集上，把 Transformer 改寫到圖上的 GT，驗證集 MAE 是 0.1400；加寬到 8,320 萬參數，反而是 0.1408。兩者都沒贏過只有 670 萬參數、加了虛擬節點的 GIN（GIN-VN，0.1395）。\n\n大連理工大學、普林斯頓大學、北京大學與微軟亞洲研究院的 Chengxuan Ying、Tianle Cai、Shengjie Luo 等人，乾脆把疑問寫進論文標題，問 Transformer 在圖上是不是真的表現很差。\n\n他們的 Graphormer 用的是標準的 Transformer，最後拿下這場比賽圖層級賽道的第一名。\n\n\n> [!TIP]\n> \n> GT 的稀疏注意力與拉普拉斯位置編碼，以及同一個月發表的 SAN，可以參考我們的筆記：\n> \n> - **[[20.12] Graph Transformer：連滿反而糟](/blog/papers/graph-learning/graph-transformer)**\n> - **[[21.06] SAN：聽見圖的形狀](/blog/papers/graph-learning/san)**\n\n\n## 定義問題\n\n作者觀察到，把 Transformer 用在圖上的嘗試不少，真正有效的多半是把 GNN 裡的某個模組（例如聚合）換成 softmax 注意力，骨架還是 GNN。\n\n問題出在哪裡？自注意力計算的是兩個節點特徵之間的語意相似度，圖的結構完全沒進入這個計算。哪個節點比較重要、兩個節點在圖上隔多遠、中間的邊是什麼類型，注意力都看不到。\n\n作者認為，關鍵在於把這些結構資訊有效地編碼進模型。\n\n## 解決問題\n\n[![原論文 Figure 1：左邊是一層自注意力，節點特徵加上中心性編碼後，經 Q、K、V 的線性投影，Q 和 K 相乘、縮放，再加上一個偏差後做 softmax，乘上 V。中間兩個 5×5 的方格分別是空間編碼與邊編碼，各自給每一對節點一個值，兩者都加到注意力分數上；下方的中心性編碼與節點特徵相加。右邊是一張五個節點的小圖，節點大小代表重要性，虛線代表節點之間的關係。](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)](/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png)\n\n*原論文 Figure 1（Ying 等人），取自 [**PDF 第 3 頁**](https://arxiv.org/pdf/2106.05234v5#page=3)，CC BY-SA 4.0。圖說未收錄。*\n\nGraphormer 加了三種編碼。一種加在節點上，另外兩種加在節點與節點之間，也就是注意力分數上。\n\n### 中心性編碼\n\n社群網路裡，追蹤者多的人通常更有影響力。作者用最簡單的度數中心性，依每個節點的入度和出度各查一個可學的向量，加到輸入的節點特徵上：\n\n$$\nh_i^{(0)}=x_i+z^-_{\\deg^-(v_i)}+z^+_{\\deg^+(v_i)}\n$$\n\n無向圖的入度和出度相同，合成一個就好。這樣 Q 和 K 裡都帶著節點的重要性。\n\n### 空間編碼\n\nTransformer 處理句子時，要另外告訴模型每個字的位置。圖沒有一條固定的軸，作者改成描述兩個節點之間的關係。\n\n他們用兩個節點之間的最短路徑距離 $\\phi(v_i,v_j)$，每一種距離對應一個可學的純量 $b_{\\phi(v_i,v_j)}$，加在注意力分數上：\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}\n$$\n\n兩個節點不連通時，距離設成一個特殊值 −1，也有自己的偏差。這組偏差在所有層之間共用。\n\n如果 $b$ 學成隨距離遞減，每個節點就會多看近的、少看遠的；但它仍然看得到全圖。\n\n### 邊編碼\n\n分子的化學鍵類型是邊的特徵。以前的做法是把邊特徵加到兩端節點上，或在聚合時一起用，邊的資訊只傳到相連的節點。\n\nGraphormer 把邊特徵也放進注意力分數。對每一對節點 $(v_i,v_j)$，找一條最短路徑 $(e_1,\\dots,e_N)$，把路上每條邊的特徵和一個可學的向量做內積，取平均：\n\n$$\nA_{ij}=\\frac{(h_iW_Q)(h_jW_K)^\\top}{\\sqrt d}+b_{\\phi(v_i,v_j)}+c_{ij},\\qquad\nc_{ij}=\\frac1N\\sum_{n=1}^N x_{e_n}\\big(w_n^E\\big)^\\top\n$$\n\n### 其他細節\n\nLayerNorm 放在注意力和前饋網路之前，也就是 Pre-LN。另外加一個和所有節點相連的特殊節點 `[VNode]`，最後一層它的表示就是整張圖的表示，作用類似 BERT 的 `[CLS]`。它和每個節點的距離都是 1，但這條連線不是真的邊，所以它的空間編碼另外用一個專屬的偏差。\n\n### 能表示哪些 GNN？\n\n作者證明，選對權重和距離函數，Graphormer 的一層可以表示 GIN、GCN、GraphSAGE 的聚合與更新。以平均聚合為例，把距離 1 的偏差設成 0、其他設成負無窮大，再讓 $W_Q=W_K=0$、$W_V=I$，softmax 就只會平均分給鄰居。\n\n拿一張五個節點的小圖驗證一次：\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\ndef build(n, edges):\n    M = np.zeros((n, n))\n    for i, j in edges:\n        M[i, j] = M[j, i] = 1\n    return M\n\ndef shortest_paths(M):\n    n = len(M)\n    dist = np.full((n, n), -1)\n    for s in range(n):                       # 從每個節點做一次廣度優先搜尋\n        dist[s, s], frontier, d = 0, [s], 0\n        while frontier:\n            d += 1\n            nxt = [v for u in frontier for v in range(n) if M[u, v] and dist[s, v] < 0]\n            for v in nxt:\n                dist[s, v] = d\n            frontier = list(dict.fromkeys(nxt))\n    return dist\n\nM = build(5, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)])\nh = rng.normal(size=(5, 3))\nspd = shortest_paths(M)\n\nbias = np.where(spd == 1, 0.0, -np.inf)          # 距離 1 給 0，其他給負無窮大\nattn = np.exp(bias) / np.exp(bias).sum(1, keepdims=True)   # W_Q = W_K = 0，只剩偏差\nout = attn @ h                                   # W_V = I\n\nmean_nbrs = np.array([h[M[v] > 0].mean(0) for v in range(5)])\nprint(\"節點 2 的注意力:\", attn[2].round(2))\nprint(\"和鄰居平均相同:\", np.allclose(out, mean_nbrs))\n```\n\n```text\n節點 2 的注意力: [0.33 0.33 0.   0.33 0.  ]\n和鄰居平均相同: True\n```\n\n節點 2 有三個鄰居，各分到三分之一，自己和距離 2 的節點分到 0。平均乘上度數就是加總，度數可以從中心性編碼取得；最大值聚合則要每個維度用一個注意力頭去挑，證明在論文附錄。\n\n### 比 WL 更強\n\n反過來，Graphormer 也看得到 WL 看不到的東西。論文附錄舉了一對圖：\n\n[![原論文附錄 Figure 2：左邊是排成兩列、每列三個節點的梯子形，四個角落的節點是粉紅色，中間兩個是藍色；右邊是兩個三角形用一座橋相連，外側四個節點是粉紅色，橋上兩個是藍色。](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)](/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png)\n\n*原論文附錄 Figure 2（Ying 等人），取自 [**PDF 第 14 頁**](https://arxiv.org/pdf/2106.05234v5#page=14)，CC BY-SA 4.0。圖說未收錄。*\n\n左邊是梯子形，右邊是兩個三角形用一座橋相連。兩張圖都有四個度數 2 的節點、兩個度數 3 的節點，WL 每一輪塗出的顏色也一樣。看最短路徑距離就不同了：\n\n```python\nfrom collections import Counter\n\ndef wl(M, rounds=3):\n    colors = [()] * len(M)                   # 一開始大家同色\n    for _ in range(rounds):                  # 新顏色 = 自己的顏色 + 鄰居顏色的多重集合\n        colors = [(colors[v], tuple(sorted(colors[u] for u in range(len(M)) if M[v, u])))\n                  for v in range(len(M))]\n    return Counter(colors)\n\nladder = build(6, [(0, 1), (1, 2), (3, 4), (4, 5), (0, 3), (1, 4), (2, 5)])\nbridge = build(6, [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)])\nprint(\"WL 的顏色統計相同:\", wl(ladder) == wl(bridge))\nfor name, G in [(\"梯子形    \", ladder), (\"兩個三角形\", bridge)]:\n    print(name, sorted({tuple(sorted(row)) for row in shortest_paths(G).tolist()}))\n```\n\n```text\nWL 的顏色統計相同: True\n梯子形     [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 2, 3)]\n兩個三角形 [(0, 1, 1, 1, 2, 2), (0, 1, 1, 2, 3, 3)]\n```\n\n每一列是某一類節點到其他節點的距離。藍色節點兩邊都是 (0, 1, 1, 1, 2, 2)；粉紅色的節點，梯子形的角落最遠只到 3 一次，三角形外側的節點有兩個距離 3 的鄰居。只要模型看得到距離，這兩張圖就分得開。\n\n作者還證明了另一件事。沒有任何編碼時，自注意力本身就能表示對全圖取平均的讀出，等於每一層都附帶一個虛擬節點。實驗中他們也觀察到，模型加深、加寬之後，訓練和驗證誤差都持續下降，沒有出現 over-smoothing。\n\n## 討論\n\n### PCQM4M-LSC\n\n主要的實驗在 PCQM4M-LSC 上。Graphormer 有兩個大小，12 層、隱藏維度 768 的完整版（4,710 萬參數），以及 6 層、512 維的小版（1,250 萬參數），在 8 張 V100 上訓練約兩天。\n\n| 模型 | 參數 | 驗證 MAE ↓ |\n| --- | ---: | ---: |\n| GCN | 2.0M | 0.1691 |\n| GIN | 3.8M | 0.1537 |\n| GIN-VN | 6.7M | 0.1395 |\n| DeeperGCN-VN | 25.5M | 0.1398 |\n| GT | 0.6M | 0.1400 |\n| GT-Wide | 83.2M | 0.1408 |\n| Graphormer（小） | 12.5M | 0.1264 |\n| Graphormer | 47.1M | **0.1234** |\n\n相對於先前最好的 GIN-VN，驗證 MAE 降了 11.5%。GT 加寬了一百多倍，表現卻沒有變好。比賽提交時，作者把 Graphormer 和另一個模型 ExpC 集成，在完整測試集上做到 0.1200，拿下圖層級賽道的第一名。\n\n### 其他資料集\n\nZINC 是最乾淨的比較。所有模型都控制在 50 萬參數左右、從頭訓練，48.9 萬參數的 GraphormerSLIM 測試 MAE 是 0.122，SAN 是 0.139，PNA 是 0.142，GT 是 0.226。\n\nOGB 的兩個分子資料集要看條件。MolPCBA 上 Graphormer 的平均精確率是 31.39%，先前最好的是 29.79%；MolHIV 上 ROC-AUC 是 80.51%，先前最好的是 79.70%。這兩個結果的 Graphormer 先在 PCQM4M-LSC 上預訓練，參數分別是 1.195 億和 4,700 萬，還用了 FLAG 資料增強對抗過擬合。為了公平，作者也列了同樣在 PCQM4M-LSC 上預訓練、再微調的 GIN-VN，分別是 29.02% 和 77.80%。\n\n### 每個編碼貢獻多少\n\n作者在 PCQM4M-LSC 上做消融，為了省計算，模型都是 12 層、只訓練 10 萬步：\n\n| 設定 | 驗證 MAE ↓ |\n| --- | ---: |\n| 不加任何結構編碼 | 0.2276 |\n| 拉普拉斯位置編碼 | 0.1483 |\n| 最短路徑的空間編碼 | 0.1427 |\n| 空間編碼＋中心性編碼 | 0.1396 |\n| 再加上邊編碼（加在節點上） | 0.1328 |\n| 再加上邊編碼（聚合時使用） | 0.1327 |\n| 再加上邊編碼（注意力偏差） | **0.1304** |\n\n沒有任何結構編碼的 Transformer，MAE 是 0.2276，和加上編碼的版本差了一大截。空間編碼勝過拉普拉斯位置編碼；邊編碼放進注意力分數，比傳統的兩種放法都好。\n\n### 代價\n\n結論裡，作者自己點出幾個限制。注意力的計算量是節點數的平方，大圖用不起；節點層級的任務需要一套取樣的方法；度數和最短路徑只是通用的選擇，特定領域可以換成更合適的編碼，例如分子裡兩個原子的 3D 距離。\n\n## 結論\n\nGraphormer 對標題的回答是，Transformer 在圖上其實不差，前提是把結構放進去。度數加在節點上，最短路徑距離和路上的邊加在注意力分數上，一個標準的 Transformer 就在當時最大的圖層級資料集上贏過所有 GNN。\n\n一年後的 GraphGPS 把 PCQM4Mv2 的驗證 MAE 再往下推，1,940 萬參數的 GPS-medium 是 0.0858，4,830 萬參數的 Graphormer 是 0.0864，兩者幾乎打平，參數卻少了一半多。\n\n如果你要在小分子上用 Transformer，先把結構編碼加上去再談別的。同樣訓練 10 萬步，拿掉它們，MAE 就從 0.13 退到 0.23。\n","imageSizes":{"/content/papers/assets/c2a2e0893f98e2ed6d93e1f4ee87040eeffbc63c4b955e6a319700c5e87f1847.png":{"width":872,"height":620},"/content/papers/assets/d1eea7bfd57f2de038ba9731f8dc42dbfbd673a11b997cc87811b35cb03be60f.png":{"width":908,"height":216}}}