{"notes":[{"id":"graph-learning/graphsage","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphSAGE","subtitle":"Sampling the Roll Call","author":"Z. Yuan","paperDate":"2017-06-07","publishedOn":"2023-10-21","world":"external/graphsage","paperTitle":"Inductive Representation Learning on Large Graphs","paperUrl":"https://arxiv.org/abs/1706.02216","bibliography":{"title":"Inductive Representation Learning on Large Graphs","authors":["William L. Hamilton","Rex Ying","Jure Leskovec"],"url":"https://arxiv.org/abs/1706.02216","pdfUrl":"https://arxiv.org/pdf/1706.02216v4","codeUrl":"https://github.com/williamleif/GraphSAGE","preprint":{"id":"1706.02216","submittedOn":"2017-06-07","url":"https://arxiv.org/abs/1706.02216"},"publication":{"venue":"NeurIPS","year":2017,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html"},"citation":"@misc{graphlearninggraphsage2017,\n  title = {{Inductive Representation Learning on Large Graphs}},\n  author = {William L. Hamilton and Rex Ying and Jure Leskovec},\n  year = {2017},\n  eprint = {1706.02216},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1706.02216}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphsage.1a0b3fd6e8b8.bib"},"thumbnail":{"src":"/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png","width":1090,"height":384},"locale":"en","path":"/content/papers/en/graph-learning/graphsage.f6ad2435eb15.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"transductive-and-inductive","text":"Transductive and Inductive","depth":3},{"id":"too-many-neighbors","text":"Too Many Neighbors","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-algorithm","text":"The Algorithm","depth":3},{"id":"sample-the-neighbors","text":"Sample the Neighbors","depth":3},{"id":"three-aggregators","text":"Three Aggregators","depth":3},{"id":"learning-without-labels","text":"Learning without Labels","depth":3},{"id":"the-connection-to-wl","text":"The Connection to WL","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"three-datasets","text":"Three Datasets","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"speed-and-sampling-tradeoffs","text":"Speed and Sampling Tradeoffs","depth":3},{"id":"what-came-next","text":"What Came Next","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gat","relation":"citing","topic":""}]},{"id":"graph-learning/gat","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GAT","subtitle":"Some Neighbors Matter More","author":"Z. Yuan","paperDate":"2017-10-30","publishedOn":"2023-11-10","world":"external/gat","paperTitle":"Graph Attention Networks","paperUrl":"https://arxiv.org/abs/1710.10903","bibliography":{"title":"Graph Attention Networks","authors":["Petar Veličković","Guillem Cucurull","Arantxa Casanova","Adriana Romero","Pietro Liò","Yoshua Bengio"],"url":"https://arxiv.org/abs/1710.10903","pdfUrl":"https://arxiv.org/pdf/1710.10903v3","codeUrl":"https://github.com/PetarV-/GAT","preprint":{"id":"1710.10903","submittedOn":"2017-10-30","url":"https://arxiv.org/abs/1710.10903"},"publication":{"venue":"ICLR","year":2018,"kind":"conference","url":"https://arxiv.org/abs/1710.10903"},"citation":"@misc{graphlearninggat2017,\n  title = {{Graph Attention Networks}},\n  author = {Petar Veličković and Guillem Cucurull and Arantxa Casanova and Adriana Romero and Pietro Liò and Yoshua Bengio},\n  year = {2017},\n  eprint = {1710.10903},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1710.10903}\n}\n","citationPath":"/content/papers/citations/graph-learning/gat.402fa57986a5.bib"},"thumbnail":{"src":"/content/papers/assets/1a68144470e25ef01f2cd6ccf405483bc7960ca1cc9f7ebf5991060fb3a92010.png","width":1124,"height":540},"locale":"en","path":"/content/papers/en/graph-learning/gat.8ad15224e4f4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"one-attention-head","text":"One Attention Head","depth":3},{"id":"multiple-heads","text":"Multiple Heads","depth":3},{"id":"cost-and-properties","text":"Cost and Properties","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"what-did-it-learn","text":"What Did It Learn?","depth":3},{"id":"the-authors-own-limitations","text":"The Authors' Own Limitations","depth":3},{"id":"static-attention","text":"Static Attention","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/gcn","relation":"referenced","topic":""},{"id":"graph-learning/graphsage","relation":"referenced","topic":""}]}],"text":"New Reddit posts arrive every minute, new YouTube videos every day. A recommender needs representations for these new nodes, ideally immediately, without retraining the whole graph.\n\nBefore 2017, mainstream node embeddings could not do this. DeepWalk and node2vec learn a separate vector for each node, like a lookup table; a new node has no entry. GCN consumes node features, but its paper trains on the whole graph, keeping the full adjacency matrix in memory at every step.\n\nWilliam Hamilton, Rex Ying and Jure Leskovec at Stanford change the learning target. Instead of learning each node's vector, learn a function that computes a vector from its neighbors. Once trained, it applies directly to any new node with features and neighbors.\n\n## Defining the Problem\n\n### Transductive and Inductive\n\nThe paper distinguishes two settings.\n\nIn the **transductive** setting, the whole graph is visible during training, though some nodes have no labels; the GCN paper's semi-supervised classification fits here. In the **inductive** setting, unseen nodes, or even entirely new graphs, arrive after training.\n\nThe authors consider induction harder. The model must learn to identify a node's role from local structure and features, rather than memorize its position. It must also place unseen nodes in a space consistent with existing ones, without seeing them during training.\n\nMost existing embeddings are transductive. Extending them to new nodes requires more optimization at test time, which is slow.\n\n### Too Many Neighbors\n\nThe second problem is scale. The paper's Reddit dataset has 232,965 posts, connecting any two commented on by the same user. Each post has 492 edges on average.\n\nIf every node examines its full two-hop neighborhood, one node can involve hundreds of thousands of others, making batch memory and runtime unpredictable.\n\n## Solving the Problem\n\n### The Algorithm\n\nGraphSAGE stands for SAmple and aggreGatE. The forward pass runs for $K$ rounds, each performing three operations per node $v$:\n\n$$\nh^k_{\\mathcal N(v)}=\\mathrm{AGGREGATE}_k\\left(\\left\\{h^{k-1}_u,\\ \\forall u\\in\\mathcal N(v)\\right\\}\\right)\n$$\n\n$$\nh^k_v=\\sigma\\left(W^k\\cdot\\mathrm{CONCAT}\\left(h^{k-1}_v,\\ h^k_{\\mathcal N(v)}\\right)\\right)\n$$\n\n$$\nh^k_v\\leftarrow h^k_v/\\left\\|h^k_v\\right\\|_2\n$$\n\nAggregate neighboring representations into one vector, **concatenate** it with the node's own representation from the previous round, multiply by a weight matrix, apply a nonlinearity, then L2-normalize. $h^0_v$ contains the input features; round $K$ outputs the node embedding.\n\nConcatenation is the main difference from GCN. GCN treats the node itself as another neighbor and takes a weighted average, sharing one $W$ among everyone. GraphSAGE separates self and neighbors, effectively giving each its own weights. The authors describe this as a kind of skip connection across depths, with substantial experimental benefits.\n\n[![Figure 1 of the paper: three steps from left to right. First sample one-hop and two-hop neighbors around the target node; then pass their features back through aggregators; finally use the aggregated representation to predict graph context and labels.](/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png)](/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png)\n\n*Figure 1 from Hamilton et al., taken from [**PDF page 2**](https://arxiv.org/pdf/1706.02216v4#page=2). The original caption is omitted.*\n\n### Sample the Neighbors\n\n$\\mathcal N(v)$ is a **fixed-size uniform sample**, resampled each round, rather than the full neighborhood. Sampling $S_i$ neighbors in round $i$ fixes batch cost at $O\\left(\\prod_{i=1}^KS_i\\right)$. The authors find $K=2$ and $S_1\\cdot S_2\\le500$ work well; experiments use $S_1=25$ and $S_2=10$.\n\nUsing Reddit's average degree, compare the sizes:\n\n```python\nS1, S2 = 25, 10                 # Sample 25 in the first layer and 10 per node in the second\navg_deg = 492                   # Reddit average degree\nprint(\"After sampling, maximum nodes per target:\", 1 + S1 + S1 * S2, \"nodes\")\nprint(\"Without sampling, two-hop upper bound:     \", 1 + avg_deg + avg_deg ** 2, \"nodes\")\n```\n\n```text\nAfter sampling, maximum nodes per target: 276 nodes\nWithout sampling, two-hop upper bound:      242557 nodes\n```\n\nThe second number is only a rough upper bound, without deduplicating neighbors, yet already exceeds the dataset's 232,965 nodes. Without sampling, computing one node's two-layer representation can amount to reading almost the entire graph.\n\nFor minibatch training, the authors first sample the required neighborhoods outward from target nodes, then compute inward layer by layer, evaluating only nodes actually needed by the recursion.\n\nNow run one layer on a six-node graph, sampling 2 neighbors per node with a mean aggregator:\n\n```python\nimport numpy as np\nnp.set_printoptions(precision=2, suppress=True)\nrng = np.random.default_rng(0)\n\nneighbors = [[1, 2], [0, 2, 3], [0, 1, 3, 4, 5], [1, 2], [2], [2]]\nh = rng.random((6, 3))                       # 6 nodes, 3 features each\nW = rng.normal(size=(4, 6))                  # 6 concatenated dimensions to 4 output dimensions\n\ndef sage_layer(h, neighbors, W, S):\n    out = []\n    for v, nbrs in enumerate(neighbors):\n        picked = rng.choice(nbrs, size=S, replace=len(nbrs) < S)   # Sample with replacement if there are too few neighbors\n        h_nb = h[picked].mean(0)                                    # Mean aggregation\n        z = np.maximum(W @ np.concatenate([h[v], h_nb]), 0)          # Concatenate self and neighbors, then multiply by W\n        out.append(z / (np.linalg.norm(z) + 1e-12))                  # L2 normalization\n    return np.array(out)\n\nh1 = sage_layer(h, neighbors, W, S=2)\nprint(h1.shape)\nprint(h1)\n```\n\n```text\n(6, 4)\n[[0.96 0.12 0.   0.24]\n [0.74 0.   0.19 0.64]\n [0.99 0.   0.   0.16]\n [0.95 0.3  0.   0.  ]\n [0.71 0.   0.   0.7 ]\n [0.81 0.   0.13 0.57]]\n```\n\nNode 2 has five neighbors but samples only two at a time. Nodes 4 and 5 have one neighbor each, so sampling repeats it. The official TensorFlow implementation similarly builds a fixed-length neighbor table per node, sampling with replacement to fill short neighborhoods, then randomly selects $S$ entries for each batch.\n\nWeights $W$ are independent of node identity. A new node with features and neighbors can therefore use the same `sage_layer` directly.\n\n### Three Aggregators\n\nNeighbors have no order, so an aggregator should ideally be permutation invariant, trainable and expressive. The authors try three:\n\n- **Mean**: elementwise averaging. They also make a GCN-style variant that omits self-concatenation and instead averages $\\{v\\}\\cup\\mathcal N(v)$ before weighting, calling it the convolutional aggregator.\n- **LSTM**: the most expressive, but inherently order-sensitive. The compromise is to shuffle neighbors randomly before each input.\n- **Pooling**: pass every neighbor independently through a fully connected layer, then take an elementwise maximum, $\\max\\left(\\left\\{\\sigma\\left(W_{\\text{pool}}h_{u}+b\\right)\\right\\}\\right)$. Replacing max with mean makes little difference, the authors report.\n\n### Learning without Labels\n\nWithout task labels, GraphSAGE uses a graph-based loss:\n\n$$\nJ_{\\mathcal G}(z_u)=-\\log\\sigma\\left(z_u^\\top z_v\\right)-Q\\cdot\\mathbb{E}_{v_n\\sim P_n(v)}\\log\\sigma\\left(-z_u^\\top z_{v_n}\\right)\n$$\n\n$v$ appears near $u$ on a fixed-length random walk, $v_n$ is a negative sample, and $Q$ is the number of negatives. This resembles DeepWalk's objective, but $z_u$ is computed from neighboring features rather than retrieved from a table. With labels, replace or augment this loss with cross-entropy.\n\n### The Connection to WL\n\nThe authors also note that setting $K$ to the number of nodes, weights to identity matrices and the aggregator to a hash function turns Algorithm 1 into the Weisfeiler–Lehman graph-isomorphism test. GraphSAGE can be viewed as a continuous WL approximation whose purpose is useful node representations rather than isomorphism testing.\n\n## Discussion\n\n### Three Datasets\n\nAll experiments use inductive settings:\n\n- **Citation network**: 302,424 Web of Science papers from six biology-related fields, 2000–2005, with average degree 9.15. Train on 2000–2004 and predict fields of 2005 papers.\n- **Reddit**: September 2014 posts from 50 large subreddits. Train on the first 20 days and predict later posts' subreddits.\n- **PPI**: protein–protein interaction graphs, one per human tissue. Train on 20 graphs, validate on 2 and test on 2, predicting 121 protein-function labels. Test graphs are entirely unseen during training.\n\n### Main Results\n\n| Method | Citation network | Reddit | PPI |\n| --- | ---: | ---: | ---: |\n| Raw features | 0.575 | 0.585 | 0.422 |\n| DeepWalk + features | 0.701 | 0.691 | — |\n| GraphSAGE-GCN | 0.772 | 0.930 | 0.500 |\n| GraphSAGE-mean | 0.820 | 0.950 | 0.598 |\n| GraphSAGE-LSTM | 0.832 | **0.954** | **0.612** |\n| GraphSAGE-pool | **0.839** | 0.948 | 0.600 |\n\nThese are micro-F1 scores for supervised variants. The GCN-style aggregator finishes last on all three datasets, nearly 0.1 behind the others on PPI. Its main difference from mean is the missing concatenation of self and neighbors.\n\nA Wilcoxon test across six settings—supervised and unsupervised on three datasets—finds LSTM, pool and mean significantly better than the GCN-style aggregator, with no significant difference between LSTM and pool. The authors acknowledge the limited statistical power of only six settings.\n\nEven the authors are somewhat surprised that LSTM, designed for sequences, works so well on unordered neighborhoods.\n\n### Speed and Sampling Tradeoffs\n\n[![Figure 2 of the paper: panel A shows Reddit training and test times, with DeepWalk testing much slower than GraphSAGE variants. Panel B varies sampled neighbors per layer on the citation network from single digits to over 60: micro-F1 rises from around 0.78 to 0.83 while runtime keeps increasing.](/content/papers/assets/ff62a5b1aa8349edd927fc6014fd713adf4d76672507efe1a7527ff6ecdc45ca.png)](/content/papers/assets/ff62a5b1aa8349edd927fc6014fd713adf4d76672507efe1a7527ff6ecdc45ca.png)\n\n*Figure 2 from Hamilton et al., taken from [**PDF page 7**](https://arxiv.org/pdf/1706.02216v4#page=7). The original caption is omitted.*\n\nOn the left, training times are similar, with GraphSAGE-LSTM slowest. At test time, however, DeepWalk must rerun random walks and SGD for new nodes, taking 100 to 500 times longer than GraphSAGE.\n\nOn the right, more sampled neighbors improve F1, but returns diminish quickly while time keeps rising. The authors also find $K=2$ improves over $K=1$ by 10% to 15% on average; increasing to $K>2$ adds only 0% to 5% while multiplying runtime by 10 to 100.\n\n### What Came Next\n\nThe paper ends by pointing to follow-up work by Chen and Zhu for better PPI results: they specifically tune GraphSAGE's hyperparameters for PPI and add dropout, layer normalization and a new sampling scheme.\n\nThe following year, Rex Ying, Hamilton and Leskovec work with Pinterest on PinSage, training with 7.5 billion examples on a graph of 3 billion nodes and deploying it as Pinterest's recommendation engine. [PinSage abstract](https://arxiv.org/abs/1806.01973v1)\n\n## Conclusion\n\nGraphSAGE makes a graph neural network learn a function rather than a lookup table. Sampling fixes batch cost, enabling minibatch training on large graphs; weighting self and neighbors separately gives a substantial gain over GCN-style averaging.\n\nFor your own large graph, start with $K=2$, $S_1=25$, $S_2=10$ and a mean aggregator as the baseline. PyTorch Geometric provides `SAGEConv` and its companion `NeighborLoader`, so sampling and minibatching need no custom implementation.\n","imageSizes":{"/content/papers/assets/88f6d9397f20155603c7ee0daef374fb944aac5fb73ed6573a573560b5b9c6aa.png":{"width":1090,"height":384},"/content/papers/assets/ff62a5b1aa8349edd927fc6014fd713adf4d76672507efe1a7527ff6ecdc45ca.png":{"width":1092,"height":336}}}