{"notes":[{"id":"graph-learning/deeper-insights","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deeper Insights","subtitle":"抹得太平了","author":"Z. Yuan","paperDate":"2018-01-22","publishedOn":"2023-11-29","paperTitle":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","paperUrl":"https://arxiv.org/abs/1801.07606","bibliography":{"title":"Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning","authors":["Qimai Li","Zhichao Han","Xiao-Ming Wu"],"url":"https://arxiv.org/abs/1801.07606","pdfUrl":"https://arxiv.org/pdf/1801.07606v1","codeUrl":null,"preprint":{"id":"1801.07606","submittedOn":"2018-01-22","url":"https://arxiv.org/abs/1801.07606"},"publication":{"venue":"AAAI","year":2018,"kind":"conference","url":"https://ojs.aaai.org/index.php/AAAI/article/view/11604"},"citation":"@misc{graphlearningdeeperinsights2018,\n  title = {{Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning}},\n  author = {Qimai Li and Zhichao Han and Xiao-Ming Wu},\n  year = {2018},\n  eprint = {1801.07606},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1801.07606}\n}\n","citationPath":"/content/papers/citations/graph-learning/deeper-insights.52354c86fac9.bib"},"thumbnail":{"src":"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png","width":1396,"height":268},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/deeper-insights.8ca58f8b9ca8.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"先和全連接網路比","text":"先和全連接網路比","depth":3},{"id":"拉普拉斯平滑","text":"拉普拉斯平滑","depth":3},{"id":"平滑太多次","text":"平滑太多次","depth":3},{"id":"太淺也不行","text":"太淺也不行","depth":3},{"id":"補救讓-gcn-多看到一些標籤","text":"補救：讓 GCN 多看到一些標籤","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/appnp","relation":"referenced","topic":""},{"id":"graph-learning/sgc","relation":"referenced","topic":""}]},{"id":"graph-learning/appnp","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"APPNP","subtitle":"記得回家","author":"Z. Yuan","paperDate":"2018-10-14","publishedOn":"2024-01-17","world":"external/appnp","paperTitle":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","paperUrl":"https://arxiv.org/abs/1810.05997","bibliography":{"title":"Predict then Propagate: Graph Neural Networks meet Personalized PageRank","authors":["Johannes Gasteiger","Aleksandar Bojchevski","Stephan Günnemann"],"url":"https://arxiv.org/abs/1810.05997","pdfUrl":"https://arxiv.org/pdf/1810.05997v6","codeUrl":"https://www.kdd.in.tum.de/ppnp","preprint":{"id":"1810.05997","submittedOn":"2018-10-14","url":"https://arxiv.org/abs/1810.05997"},"publication":{"venue":"ICLR","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1810.05997"},"citation":"@misc{graphlearningappnp2018,\n  title = {{Predict then Propagate: Graph Neural Networks meet Personalized PageRank}},\n  author = {Johannes Gasteiger and Aleksandar Bojchevski and Stephan Günnemann},\n  year = {2018},\n  eprint = {1810.05997},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1810.05997}\n}\n","citationPath":"/content/papers/citations/graph-learning/appnp.eae593f21b4d.bib"},"thumbnail":{"src":"/content/papers/assets/9ad06d29a0d48e1de136b4cd8560f14defc55b0aaf289f07b8319c40f033b844.png","width":908,"height":370},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/appnp.d5272f045f40.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"疊到無限多層會怎樣","text":"疊到無限多層會怎樣","depth":3},{"id":"解決問題","text":"解決問題","depth":2},{"id":"personalized-pagerank","text":"Personalized PageRank","depth":3},{"id":"ppnp先預測再傳播","text":"PPNP：先預測，再傳播","depth":3},{"id":"appnp用冪迭代近似","text":"APPNP：用冪迭代近似","depth":3},{"id":"和-gcn-差在哪","text":"和 GCN 差在哪","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"嚴格的評估","text":"嚴格的評估","depth":3},{"id":"主要結果","text":"主要結果","depth":3},{"id":"要傳幾步多常回家","text":"要傳幾步、多常回家","depth":3},{"id":"傳播用在哪裡","text":"傳播用在哪裡","depth":3},{"id":"標籤很少的時候","text":"標籤很少的時候","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""}]},{"id":"graph-learning/sgc","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"SGC","subtitle":"線性就夠了？","author":"Z. Yuan","paperDate":"2019-02-19","publishedOn":"2024-02-09","world":"external/sgc","paperTitle":"Simplifying Graph Convolutional Networks","paperUrl":"https://arxiv.org/abs/1902.07153","bibliography":{"title":"Simplifying Graph Convolutional Networks","authors":["Felix Wu","Tianyi Zhang","Amauri Holanda de Souza","Christopher Fifty","Tao Yu","Kilian Q. Weinberger"],"url":"https://arxiv.org/abs/1902.07153","pdfUrl":"https://arxiv.org/pdf/1902.07153v2","codeUrl":"https://github.com/Tiiiger/SGC","preprint":{"id":"1902.07153","submittedOn":"2019-02-19","url":"https://arxiv.org/abs/1902.07153"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/wu19e.html"},"citation":"@misc{graphlearningsgc2019,\n  title = {{Simplifying Graph Convolutional Networks}},\n  author = {Felix Wu and Tianyi Zhang and Amauri Holanda de Souza and Christopher Fifty and Tao Yu and Kilian Q. Weinberger},\n  year = {2019},\n  eprint = {1902.07153},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.07153}\n}\n","citationPath":"/content/papers/citations/graph-learning/sgc.5b3815dbf17c.bib"},"thumbnail":{"src":"/content/papers/assets/42da0472259826c266fe1b1b617041ef7cb3aa49eda3558ac91dcd15aa5272e0.png","width":1366,"height":612},"locale":"zh-hant","path":"/content/papers/zh-hant/graph-learning/sgc.44503a2cc380.md","headings":[{"id":"定義問題","text":"定義問題","depth":2},{"id":"解決問題","text":"解決問題","depth":2},{"id":"拿掉非線性","text":"拿掉非線性","depth":3},{"id":"從頻域看它是一個低通濾波器","text":"從頻域看：它是一個低通濾波器","depth":3},{"id":"討論","text":"討論","depth":2},{"id":"引用網路與-reddit","text":"引用網路與 Reddit","depth":3},{"id":"快多少","text":"快多少","depth":3},{"id":"下游任務","text":"下游任務","depth":3},{"id":"結論","text":"結論","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gin","relation":"referenced","topic":""}]}],"text":"GCN 在 2016 年底一推出，就在三個引用網路的半監督分類上大幅領先。但它為什麼有效，論文本身沒有說得很清楚；推導從頻域圖卷積出發，一路近似，最後得到一行很短的傳播規則。\n\n香港理工大學的 Qimai Li、Zhichao Han 與 Xiao-Ming Wu 想把這一行公式拆開來看。它到底在做什麼？什麼情況下會失靈？\n\n他們的答案是，GCN 的圖卷積是一種拉普拉斯平滑；平滑一次有用，平滑太多次，所有節點都會變得一樣。這個現象叫做 **over-smoothing**，後來 APPNP、over-squashing 等論文談到 GNN 疊不深時，引用的就是這篇。\n\n## 定義問題\n\n作者點出 GCN 的兩個實際問題。\n\n第一，GCN 是淺層的，論文用的是兩層。兩層 GCN 只看得到兩步以內的鄰居；標籤很少的時候，大部分節點身邊根本沒有任何有標籤的節點，標籤資訊傳不到整張圖。\n\n第二，GCN 的訓練依賴驗證集。原論文每類只用 20 個標籤訓練，例如 Cora 一共 140 個，卻另外用了 500 個有標籤的節點做驗證、決定何時停止訓練。驗證集比訓練集還大，這違背了半監督學習的初衷，也讓它和不需要驗證集的方法比較時不太公平。\n\n作者想先弄清楚 GCN 有效的原因，再針對這兩個問題提出改進。\n\n## 解決問題\n\n### 先和全連接網路比\n\nGCN 和全連接網路（FCN）唯一的差別，是在特徵矩陣的左邊多乘了一個 $\\hat A=\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}$。作者在 Cora 上，每類 20 個標籤，比較兩者：\n\n| 一層 FCN | 兩層 FCN | 一層 GCN | 兩層 GCN |\n| ---: | ---: | ---: | ---: |\n| 53.1% | 55.9% | 70.8% | 79.8% |\n\n只有一層時，GCN 就比 FCN 高了將近 18 個百分點。差別全在那個 $\\hat A$。\n\n### 拉普拉斯平滑\n\n把一層 GCN 拆成兩步。先做圖卷積 $Y=\\hat AX$，再接一層全連接。關鍵顯然在第一步。\n\n拉普拉斯平滑是圖形學裡平滑網格的老方法。對每個特徵通道，每個節點把自己和鄰居的值混在一起：\n\n$$\n\\hat y_i=(1-\\gamma)x_i+\\gamma\\sum_j\\frac{\\tilde a_{ij}}{d_i}x_j,\\qquad 0<\\gamma\\le1\n$$\n\n寫成矩陣是 $\\hat Y=(I-\\gamma\\tilde D^{-1}\\tilde L)X$，其中 $\\tilde L=\\tilde D-\\tilde A$。令 $\\gamma=1$，就是只用鄰居（含自己）的平均 $\\tilde D^{-1}\\tilde AX$。再把 $\\tilde D^{-1}\\tilde L$ 換成對稱正規化的 $\\tilde D^{-1/2}\\tilde L\\tilde D^{-1/2}$，得到的正好是 GCN 的 $\\tilde D^{-1/2}\\tilde A\\tilde D^{-1/2}X$。\n\n所以 GCN 的圖卷積是一種**對稱拉普拉斯平滑**。它讓每個節點的特徵變成自己和鄰居的加權平均；在引用網路這類圖上，相連的節點傾向同一類，平均之後，同類的節點變得更像，後面的分類器就好做多了。作者認為，這就是 GCN 有效的關鍵。\n\n### 平滑太多次\n\n如果平滑是好事，多疊幾層呢？\n\n作者用 Zachary 空手道俱樂部做實驗。34 位成員、78 條邊，後來分裂成兩派。GCN 的權重隨機初始化、不訓練，節點特徵用 one-hot，隱藏層 16 維，輸出 2 維，畫出 1 到 5 層的輸出：\n\n[![原論文 Figure 2：五張散布圖，分別是 1 到 5 層未訓練 GCN 輸出的二維嵌入，黃點與黑點代表兩派成員。1 層時兩色混在一起；2 層時大致分開；3 到 5 層時兩色又逐漸混在一起，5 層時點都擠成一條細線。](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)](/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png)\n\n*原論文 Figure 2（Li 等人），取自 [**PDF 第 4 頁**](https://arxiv.org/pdf/1801.07606v1#page=4)。主圖說未收錄。*\n\n一層時分得不好，兩層時兩派大致分開，再往上疊，兩派又混在一起。作者說，這個資料集很小，兩派之間的連線也不少，所以混得特別快。\n\n我們可以把隨機權重拿掉，只看平滑本身。把每個節點的 one-hot 特徵平滑 $k$ 次，比較同一派和不同派成員之間的餘弦相似度：\n\n```python\nimport networkx as nx\nimport numpy as np\n\nG = nx.karate_club_graph()\nA = nx.to_numpy_array(G, weight=None) + np.eye(34)        # 34 位成員、78 條邊，加上自環\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nclub = np.array([G.nodes[v][\"club\"] == \"Mr. Hi\" for v in G])   # 俱樂部後來分裂成兩派\nsame = club[:, None] == club[None, :]\n\nfor k in (1, 2, 4, 16, 64):\n    H = np.linalg.matrix_power(A_hat, k)                   # 特徵用單位矩陣：H = Â^k I\n    Hn = H / np.linalg.norm(H, axis=1, keepdims=True)\n    S = Hn @ Hn.T                                          # 兩兩之間的餘弦相似度\n    print(f\"k={k:>2}  同一派 {S[same].mean():.3f}   不同派 {S[~same].mean():.3f}\")\n```\n\n```text\nk= 1  同一派 0.273   不同派 0.054\nk= 2  同一派 0.542   不同派 0.179\nk= 4  同一派 0.818   不同派 0.424\nk=16  同一派 0.990   不同派 0.942\nk=64  同一派 1.000   不同派 1.000\n```\n\n平滑幾次之後，同一派的成員確實比不同派的更像，這是 GCN 有用的地方。但不同派的相似度也一路上升，到了 64 次，兩者都是 1.000，所有人的特徵指向同一個方向，兩派再也分不開。\n\n作者把這件事寫成定理。只要圖沒有二部的連通分量，反覆套用拉普拉斯平滑，每個連通分量裡的節點特徵都會收斂到同樣的值；對稱版本則收斂到和度數平方根成正比的值。加了自環的圖一定不是二部圖，所以 GCN 疊夠多層一定會走到這一步。\n\n### 太淺也不行\n\n不能疊深，那就只好淺。但淺有淺的問題。\n\n[![原論文 Figure 1：橫軸是 Cora 上的訓練集比例，從 0.01 到 0.05，縱軸是測試準確率。標籤傳播的藍線在 0.62 到 0.70 之間緩慢上升；有驗證集的 GCN（橘線）從約 0.62 升到 0.79；沒有驗證集的 GCN（綠線）在 0.01 時只有約 0.54；作者方法的紅線始終最高，從約 0.70 升到 0.82。](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)](/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png)\n\n*原論文 Figure 1（Li 等人），取自 [**PDF 第 2 頁**](https://arxiv.org/pdf/1801.07606v1#page=2)。圖說未收錄。*\n\n標籤越少，GCN 掉得越快，比只用圖結構的標籤傳播掉得還快。作者的解釋是，兩層 GCN 只能把標籤資訊傳到兩步以內，看不到整張圖的結構；而標籤傳播本來就是沿著整張圖擴散的。沒有驗證集時，GCN 掉得更多。\n\n### 補救：讓 GCN 多看到一些標籤\n\n作者的辦法不改模型，改訓練資料，替 GCN 擴充標籤集。\n\n- **Co-training**：用一個能看到整張圖的隨機遊走模型（partially absorbing random walks），替每一類找出離已知標籤最近的幾個節點，加進訓練集。\n- **Self-training**：先訓練一個 GCN，挑出它最有把握的預測，加進訓練集，再繼續訓練。這在圖有很多小的孤立分量、隨機遊走傳不過去時特別有用。\n- **Union** 與 **Intersection**：把兩種方法找到的標籤取聯集或交集。聯集讓標籤更多樣；交集只留兩者都有把握的，比較準。\n\n這四種方法都直接在擴充後的標籤集上訓練，不需要額外的驗證集。\n\n## 討論\n\n作者在 Cora、CiteSeer、PubMed 上，把標籤比例降到遠低於原 GCN 論文的設定。以 Cora 為例：\n\n| 標籤比例 | 0.5% | 1% | 2% | 3% | 4% | 5% |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| 標籤傳播 | 56.4 | 62.3 | 65.4 | 67.5 | 69.0 | 70.2 |\n| ChebNet | 38.0 | 52.0 | 62.4 | 70.8 | 74.1 | 77.6 |\n| GCN，無驗證集 | 42.6 | 56.9 | 67.8 | 74.9 | 77.6 | 79.3 |\n| GCN，有驗證集 | 50.9 | 62.3 | 72.2 | 76.5 | 78.4 | 79.7 |\n| Union | **58.5** | **69.9** | **75.9** | **78.5** | **80.4** | **81.7** |\n\n數字是準確率（%）。標籤只有 0.5% 時，沒有驗證集的 GCN 只有 42.6%，連標籤傳播的 56.4% 都比不上；Union 是 58.5%。作者統計，在 Cora 和 CiteSeer 上標籤比例 1% 時，他們的方法比沒有驗證集的 GCN 高 23% 和 28%；PubMed 上標籤比例 0.05% 時，比沒有驗證集的 GCN 高 37%。\n\n標籤夠多時，差距就消失了，代表給定的標籤已經足以訓練出好的 GCN。ChebNet 在多數設定下表現不好，作者推測是過擬合。\n\n另一個值得一提的數字是，Union 在 Cora 上只用 2% 的標籤（54 個），就勝過原 GCN 論文表中所有使用 140 個標籤的基準方法。\n\n## 結論\n\n這篇論文把 GCN 的圖卷積解釋成對稱拉普拉斯平滑。平滑讓同類的節點更像，這是 GCN 有效的原因；平滑太多次，所有節點會收斂到只和度數有關的同一個樣子，這就是 over-smoothing。兩件事是同一個運算的兩面。\n\nGCN 因此只能淺，而淺的 GCN 在標籤很少時又傳不遠。作者用隨機遊走和自我訓練替它擴充標籤，在極少標籤的設定下拉開了明顯的差距。\n\n兩年多後，Alon 與 Yahav 在討論 over-squashing 時寫道，GNN 疊不深的公認解釋就是 over-smoothing。同一年稍晚的 APPNP 和隔年的 SGC，一個想辦法讓傳播不會糊掉，一個乾脆只留下平滑。\n\n\n> [!TIP]\n> \n> 這兩條後續的路線，可以參考我們的筆記：\n> \n> - **[[18.10] APPNP：記得回家](/blog/papers/graph-learning/appnp)**\n> - **[[19.02] SGC：線性就夠了？](/blog/papers/graph-learning/sgc)**\n\n\n下次你的 GCN 越疊越差，先別急著怪優化器。把中間層的特徵拿出來算一下兩兩相似度，可能早就抹平了。\n","imageSizes":{"/content/papers/assets/2a13880b48fae6456e19c622cee1d7c7907f8c31d6890722d4980a381fddb06a.png":{"width":1396,"height":268},"/content/papers/assets/86813b55f0a68265d57576c54dda7b6262bacae68112a45c8924d330103ed687.png":{"width":628,"height":302}}}