{"notes":[{"id":"graph-learning/graph-transformer","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Graph Transformer","subtitle":"連滿反而糟","author":"Z. Yuan","paperDate":"2020-12-17","publishedOn":"2024-03-13","paperTitle":"A Generalization of Transformer Networks to Graphs","paperUrl":"https://arxiv.org/abs/2012.09699","bibliography":{"title":"A Generalization of Transformer Networks to Graphs","authors":["Vijay Prakash Dwivedi","Xavier Bresson"],"url":"https://arxiv.org/abs/2012.09699","pdfUrl":"https://arxiv.org/pdf/2012.09699v2","codeUrl":"https://github.com/graphdeeplearning/graphtransformer","preprint":{"id":"2012.09699","submittedOn":"2020-12-17","url":"https://arxiv.org/abs/2012.09699"},"publication":{"venue":"AAAI Workshop on Deep Learning on Graphs","shortVenue":"DLG-AAAI","year":2021,"kind":"workshop","url":"https://arxiv.org/abs/2012.09699v2"},"citation":"@misc{graphlearninggraphtransformer2020,\n  title = {{A Generalization of Transformer Networks to Graphs}},\n  author = {Vijay Prakash Dwivedi and Xavier Bresson},\n  year = {2020},\n  eprint = {2012.09699},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2012.09699}\n}\n","citationPath":"/content/papers/citations/graph-learning/graph-transformer.4e2f841bbbf0.bib"},"thumbnail":{"src":"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png","width":1324,"height":1000},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graph-transformer.1a6743ef39d8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"只在鄰居之間算注意力","text":"只在鄰居之間算注意力","depth":3},{"id":"拉普拉斯位置編碼","text":"拉普拉斯位置編碼","depth":3},{"id":"邊特徵與正規化","text":"邊特徵與正規化","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"實驗設定","text":"實驗設定","depth":3},{"id":"全連接會怎樣","text":"全連接會怎樣？","depth":3},{"id":"位置編碼與-batchnorm","text":"位置編碼與 BatchNorm","depth":3},{"id":"和-gnn-比","text":"和 GNN 比","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/san","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SAN","subtitle":"聽見圖的形狀","author":"Z. Yuan","paperDate":"2021-06-07","publishedOn":"2024-03-22","world":"external/san","paperTitle":"Rethinking Graph Transformers with Spectral Attention","paperUrl":"https://arxiv.org/abs/2106.03893","bibliography":{"title":"Rethinking Graph Transformers with Spectral Attention","authors":["Devin Kreuzer","Dominique Beaini","William L. Hamilton","Vincent Létourneau","Prudencio Tossou"],"url":"https://arxiv.org/abs/2106.03893","pdfUrl":"https://arxiv.org/pdf/2106.03893v3","codeUrl":"https://github.com/DevinKreuzer/SAN","preprint":{"id":"2106.03893","submittedOn":"2021-06-07","url":"https://arxiv.org/abs/2106.03893"},"publication":{"venue":"NeurIPS","year":2021,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2021/hash/b4fd1d2cb085390fbbadae65e07876a7-Abstract.html"},"citation":"@misc{graphlearningsan2021,\n  title = {{Rethinking Graph Transformers with Spectral Attention}},\n  author = {Devin Kreuzer and Dominique Beaini and William L. Hamilton and Vincent Létourneau and Prudencio Tossou},\n  year = {2021},\n  eprint = {2106.03893},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.03893}\n}\n","citationPath":"/content/papers/citations/graph-learning/san.e69e469c3b91.bib"},"thumbnail":{"src":"/content/papers/assets/239f3ac016f803fed4fedec5cad345c48bc6b97791249f0e5cfdf2efd433b5c2.png","width":1280,"height":840},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/san.e84a43ca7dce.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"學出來的位置編碼","text":"學出來的位置編碼","depth":3},{"id":"全連接但分開兩套注意力","text":"全連接，但分開兩套注意力","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"gamma-該設多大","text":"\\gamma 該設多大？","depth":3},{"id":"全連接和-lpe-各自的貢獻","text":"全連接和 LPE 各自的貢獻","depth":3},{"id":"和當時最好的模型比","text":"和當時最好的模型比","depth":3},{"id":"代價","text":"代價","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"圖上的 GPS","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/graphgps.9db287ae3c36.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"位置與結構","text":"位置與結構","depth":3},{"id":"為什麼訊息傳遞需要它們","text":"為什麼訊息傳遞需要它們","depth":3},{"id":"先傳遞再注意會先丟掉資訊","text":"先傳遞再注意，會先丟掉資訊","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"三份材料","text":"三份材料","depth":3},{"id":"一層-gps","text":"一層 GPS","depth":3},{"id":"線性的複雜度","text":"線性的複雜度","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"消融兩條路各自的份量","text":"消融：兩條路各自的份量","depth":3},{"id":"和其他模型比","text":"和其他模型比","depth":3},{"id":"一年後的重新評估","text":"一年後的重新評估","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"Transformer 處理一句話時，每個字都會看其他所有的字。換個角度看，這就是把一句話當成一張全連接的圖，每個字是一個節點，任兩個字之間都有一條邊。\n\n既然句子已經是圖，真正的圖能不能直接丟進 Transformer？\n\n南洋理工大學的 Vijay Prakash Dwivedi 與 Xavier Bresson 認為不能照搬。句子沒有現成的連接關係，通常也只有幾十、幾百個字，全連接合理，也算得動；圖卻自帶結構，節點還可能多到上百萬個，直接連滿既浪費了這份結構，也算不起。\n\n他們在 2020 年底提出的 Graph Transformer（下文簡稱 GT），是把 Transformer 改寫到任意圖上的早期嘗試，半年後的 SAN、再隔一年的 GraphGPS 都拿它當比較的起點。\n\n\n> [!TIP]\n> \n> 後來讓全連接重新站起來的 SAN，以及把兩種做法放進同一層的 GraphGPS，可以參考我們的筆記：\n> \n> - **[[21.06] SAN：聽見圖的形狀](/blog/papers/graph-learning/san)**\n> - **[[22.05] GraphGPS：圖上的 GPS](/blog/papers/graph-learning/graphgps)**\n\n\n## 定義問題\n\n作者把問題拆成兩件事。\n\n**稀疏性**。更早的圖 Transformer（Li 等人 2019）讓每個節點看所有節點，以取得全域資訊。作者認為這樣丟掉了圖的稀疏連接，而稀疏連接本身就是很好的歸納偏差；全域資訊可以改用位置編碼、虛擬節點等方法補。\n\n**位置編碼**。句子裡的字有先後，Transformer 用正弦波替每個位置編碼。圖沒有標準的節點順序，還有對稱性，很難定義一個唯一的位置。作者認為，只在鄰居之間算注意力的 GAT 表現不夠好，原因之一就是缺了位置資訊。\n\n這篇要驗證的是，只看鄰居的注意力，加上合適的位置編碼，能不能做出一個通用又好用的圖 Transformer。\n\n## 解決問題\n\n[![原論文 Figure 1：兩張方塊圖。左邊是 Graph Transformer 層，節點特徵加上拉普拉斯位置編碼 λ 後，經過 Q、K、V 投影、相乘、縮放、softmax 與加總，再接 Add&Norm、FFN、Add&Norm，重複 L 層；右邊是帶邊特徵的版本，多了一條邊特徵 e_ij 的路徑，注意力分數和 E 投影後的邊特徵相乘，邊的表示也有自己的 FFN 與正規化。](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)](/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png)\n\n*原論文 Figure 1（Dwivedi 與 Bresson），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/2012.09699v2#page=2)。圖說未收錄。*\n\n左邊是基本版，右邊是加上邊特徵的版本。和原本的 Transformer 相比，改動有三處。\n\n### 只在鄰居之間算注意力\n\n一層 GT 的注意力和 Transformer 幾乎一樣，差別只在加總的範圍：\n\n$$\n\\hat h_i^{\\ell+1}=O_h^\\ell\\,\\Big\\Vert_{k=1}^{H}\\Big(\\sum_{j\\in\\mathcal N_i}w_{ij}^{k,\\ell}V^{k,\\ell}h_j^\\ell\\Big),\\qquad\nw_{ij}^{k,\\ell}=\\mathrm{softmax}_j\\Big(\\frac{Q^{k,\\ell}h_i^\\ell\\cdot K^{k,\\ell}h_j^\\ell}{\\sqrt{d_k}}\\Big)\n$$\n\n$\\mathcal N_i$ 是節點 $i$ 的鄰居，$\\Vert$ 表示把 $H$ 個注意力頭的結果接起來。Transformer 對所有位置加總，GT 只對鄰居加總。之後照樣接殘差、正規化和前饋網路。\n\n### 拉普拉斯位置編碼\n\n位置編碼用的是拉普拉斯矩陣的特徵向量：\n\n$$\n\\Delta=I-D^{-1/2}AD^{-1/2}=U^\\top\\Lambda U\n$$\n\n作者取最小的 $k$ 個非平凡特徵向量，也就是跳過特徵值 0 的那一條，每個節點在這 $k$ 條向量上的值，就是它的位置編碼 $\\lambda_i$。這組數字經過一層線性投影，只在輸入層加到節點特徵上，中間的層不再加。\n\n為什麼是特徵向量？作者的理由是，它們是 NLP 正弦位置編碼在圖上的推廣，能把距離的資訊帶進來，相近的節點拿到相近的編碼。\n\n拿一張兩個三角形用一座橋相連的小圖試算。A、B、C 在左邊，D、E、F 在右邊，C 和 D 是橋，取 $k=2$：\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\n\nnames = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4), (3, 5), (4, 5)]\nA = np.zeros((6, 6))\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\n\nd = A.sum(1)\nL = np.eye(6) - A / np.sqrt(np.outer(d, d))      # Δ = I - D^-1/2 A D^-1/2\nlam, U = np.linalg.eigh(L)                      # 特徵值由小到大\nU = U * np.sign(U[0])                           # 我們自己定的規矩：讓 A 那一格為正\n\npe = U[:, 1:3]                                  # 跳過特徵值 0，取接下來的 2 條\nprint(\"特徵值:\", lam[1:3])\nfor name, row in zip(names, pe):\n    print(name, row)\n```\n\n```text\n特徵值: [0.2  1.17]\nA [0.45 0.33]\nB [0.45 0.33]\nC [ 0.32 -0.53]\nD [-0.32 -0.53]\nE [-0.45  0.33]\nF [-0.45  0.33]\n```\n\n第一維把左右兩群分開，左邊三角形為正、右邊為負；第二維把橋上的 C、D 和外側的四個節點分開。A 和 B 拿到一模一樣的編碼，因為它們在圖上本來就對稱，只看結構的編碼分不出它們。\n\n特徵向量還有一個麻煩，正負號是任意的。上面那行 `np.sign` 是我們自己定的規矩；把整條向量乘上 −1，它照樣是特徵向量：\n\n```python\nv = pe[:, 0]\nprint(\"原本的向量:\", np.allclose(L @ v, lam[1] * v))\nprint(\"翻號之後  :\", np.allclose(L @ -v, lam[1] * -v))\n```\n\n```text\n原本的向量: True\n翻號之後  : True\n```\n\n同一張圖，可能拿到兩種相反的位置編碼；取 $k$ 條向量，就有 $2^k$ 種組合。換一張圖，「讓第一個節點為正」這種規矩也未必說得出道理。GT 的做法是訓練時隨機翻轉正負號，讓模型學著不在乎。\n\n### 邊特徵與正規化\n\n化學分子的鍵類型、知識圖譜裡的關係，這些資訊放在邊上。GT 的邊特徵版本先算出鄰居之間的注意力分數，再和投影後的邊特徵 $E^{k,\\ell}e_{ij}^\\ell$ 相乘，然後才做 softmax；邊的表示也有自己的前饋網路，一層一層往上更新。\n\n最後一個改動是正規化。作者把 Transformer 的 LayerNorm 換成 BatchNorm，發現訓練更快，泛化也更好。\n\n## 討論\n\n### 實驗設定\n\n作者用 Benchmarking GNNs 的三個資料集：\n\n- **ZINC**：分子的圖回歸，取 1.2 萬個分子的子集，預測溶解度相關的性質，指標是 MAE，越低越好。鍵的類型是邊特徵，所以用帶邊特徵的 GT。\n- **PATTERN**：用隨機區塊模型生成的 1.4 萬張圖，每個節點分成兩類。\n- **CLUSTER**：同樣是生成的 1.2 萬張圖，每個節點要分到 6 個群之一。\n\n模型都是 10 層、8 個注意力頭，參數控制在 50 萬左右，每組實驗跑 4 個亂數種子。\n\n### 全連接會怎樣？\n\n作者把同一個模型分別放在原本的圖，以及把所有節點兩兩連起來的全連接圖上。用 BatchNorm 和拉普拉斯位置編碼時：\n\n| 資料集 | 只看鄰居 | 看所有節點 |\n| --- | ---: | ---: |\n| ZINC（MAE ↓） | 0.226 | 0.598 |\n| CLUSTER（準確率 ↑） | 73.17% | 27.12% |\n| PATTERN（準確率 ↑） | 84.81% | 54.94% |\n\n三個資料集都大幅變差。PATTERN 只有兩類，54.94% 離亂猜不遠；CLUSTER 有 6 類，27.12% 也只比亂猜的 16.7% 好一些。ZINC 的全連接版本沒有用邊特徵，因為作者要看的就是完全沒有稀疏結構的情況。\n\n作者的結論是，在結構任意的圖上，稀疏連接是關鍵的歸納偏差。\n\n### 位置編碼與 BatchNorm\n\n作者比較了 LayerNorm／BatchNorm 與有無位置編碼的四種組合，三個資料集上最好的都是 BatchNorm 加拉普拉斯位置編碼。有些組合的變異大得嚇人，例如 PATTERN 上用 LayerNorm、不加位置編碼，四個種子的準確率是 73.14% ± 13.63%。\n\n作者也拿 Graph-BERT 用的 WL 位置編碼來比：\n\n| 位置編碼 | ZINC（MAE ↓） | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| 不用 | 0.264 | 72.14% | 83.95% |\n| 拉普拉斯 | **0.226** | **73.17%** | **84.81%** |\n| WL（Graph-BERT） | 0.267 | 70.79% | 75.49% |\n\nWL 位置編碼在 PATTERN 上的訓練準確率是 97.03%，測試只有 75.49%，作者認為它在這類生成的圖上容易過擬合。\n\n### 和 GNN 比\n\n最後和同樣 50 萬參數的 GNN 比：\n\n| 模型 | ZINC（MAE ↓） | CLUSTER | PATTERN |\n| --- | ---: | ---: | ---: |\n| GCN | 0.367 | 68.50% | 71.89% |\n| GAT | 0.384 | 70.59% | 78.27% |\n| GatedGCN | **0.214** | **76.08%** | **86.51%** |\n| GT | 0.226 | 73.17% | 84.81% |\n\nGT 明顯贏過 GCN 和同樣用注意力的 GAT，但三個資料集都還差 GatedGCN 一點。作者把 GT 定位成一個新的、更強的注意力式 GNN 基準，而不是當時最好的模型。\n\n## 結論\n\nGT 對「圖要不要連滿」的回答是先別連。只看鄰居的注意力、拉普拉斯位置編碼和 BatchNorm 一起用時效果最好，全連接的版本在三個資料集上都大幅落後。\n\n這個結論沒有維持太久。半年後的 SAN 讓全連接重新站起來，前提是位置編碼做得更講究，而且讓模型自己調配補上的連線分到多少比重；再隔一年，Dwivedi 自己也是作者之一的 GraphGPS，乾脆把只看鄰居的訊息傳遞和看所有節點的注意力放進同一層。\n\n拿拉普拉斯特徵向量當位置編碼之前，先想好正負號怎麼處理。不然同一張圖，每次算出來的編碼都可能不一樣。\n","imageSizes":{"/content/papers/assets/0540122812c6e6914f22240411cf2d172a94764bf8a0e6f6ab2cdae7c618f9fb.png":{"width":1324,"height":1000}}}