{"notes":[{"id":"graph-learning/mpnn","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"MPNN","subtitle":"So They Were All Passing Messages","author":"Z. Yuan","paperDate":"2017-04-04","publishedOn":"2023-10-11","world":"external/mpnn","paperTitle":"Neural Message Passing for Quantum Chemistry","paperUrl":"https://arxiv.org/abs/1704.01212","bibliography":{"title":"Neural Message Passing for Quantum Chemistry","authors":["Justin Gilmer","Samuel S. Schoenholz","Patrick F. Riley","Oriol Vinyals","George E. Dahl"],"url":"https://arxiv.org/abs/1704.01212","pdfUrl":"https://arxiv.org/pdf/1704.01212v2","codeUrl":null,"preprint":{"id":"1704.01212","submittedOn":"2017-04-04","url":"https://arxiv.org/abs/1704.01212"},"publication":{"venue":"ICML","year":2017,"kind":"conference","url":"https://proceedings.mlr.press/v70/gilmer17a.html"},"citation":"@misc{graphlearningmpnn2017,\n  title = {{Neural Message Passing for Quantum Chemistry}},\n  author = {Justin Gilmer and Samuel S. Schoenholz and Patrick F. Riley and Oriol Vinyals and George E. Dahl},\n  year = {2017},\n  eprint = {1704.01212},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1704.01212}\n}\n","citationPath":"/content/papers/citations/graph-learning/mpnn.5c9aab7ce227.bib"},"thumbnail":{"src":"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png","width":628,"height":388},"locale":"en","path":"/content/papers/en/graph-learning/mpnn.15429a1e914f.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-message-passing-framework","text":"The Message Passing Framework","depth":3},{"id":"which-variations-did-the-authors-try","text":"Which Variations Did the Authors Try?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"main-results-on-qm9","text":"Main Results on QM9","depth":3},{"id":"without-spatial-information","text":"Without Spatial Information","depth":3},{"id":"virtual-edges-and-the-master-node","text":"Virtual Edges and the Master Node","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/over-squashing","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/over-squashing","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Over-squashing","subtitle":"No Room Left","author":"Z. Yuan","paperDate":"2020-06-09","publishedOn":"2024-03-03","paperTitle":"On the Bottleneck of Graph Neural Networks and its Practical Implications","paperUrl":"https://arxiv.org/abs/2006.05205","bibliography":{"title":"On the Bottleneck of Graph Neural Networks and its Practical Implications","authors":["Uri Alon","Eran Yahav"],"url":"https://arxiv.org/abs/2006.05205","pdfUrl":"https://arxiv.org/pdf/2006.05205v4","codeUrl":"https://github.com/tech-srl/bottleneck","preprint":{"id":"2006.05205","submittedOn":"2020-06-09","url":"https://arxiv.org/abs/2006.05205"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2006.05205"},"citation":"@misc{graphlearningoversquashing2020,\n  title = {{On the Bottleneck of Graph Neural Networks and its Practical Implications}},\n  author = {Uri Alon and Eran Yahav},\n  year = {2020},\n  eprint = {2006.05205},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2006.05205}\n}\n","citationPath":"/content/papers/citations/graph-learning/over-squashing.af346c012204.bib"},"thumbnail":{"src":"/content/papers/assets/c3454ca38b6e53b6467ff66db7867788efed8d8a71eec3c6e7f754e586a99d4d.png","width":1078,"height":356},"locale":"en","path":"/content/papers/en/graph-learning/over-squashing.146d06e9e263.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"problem-radius","text":"Problem Radius","depth":3},{"id":"the-rnn-bottleneck","text":"The RNN Bottleneck","depth":3},{"id":"looking-only-at-propagation-shares","text":"Looking Only at Propagation Shares","depth":3},{"id":"a-deliberately-constructed-task","text":"A Deliberately Constructed Task","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-fully-adjacent-final-layer","text":"A Fully Adjacent Final Layer","depth":3},{"id":"how-large-must-the-vector-be","text":"How Large Must the Vector Be?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"tree-neighborsmatch","text":"Tree-NeighborsMatch","depth":3},{"id":"qm9","text":"QM9","depth":3},{"id":"biological-datasets","text":"Biological Datasets","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/deeper-insights","relation":"referenced","topic":""},{"id":"graph-learning/graphgps","relation":"referenced","topic":""}]},{"id":"graph-learning/graphgps","category":"graph-learning","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"GraphGPS","subtitle":"GPS for Graphs","author":"Z. Yuan","paperDate":"2022-05-25","publishedOn":"2025-09-09","world":"external/graphgps","paperTitle":"Recipe for a General, Powerful, Scalable Graph Transformer","paperUrl":"https://arxiv.org/abs/2205.12454","bibliography":{"title":"Recipe for a General, Powerful, Scalable Graph Transformer","authors":["Ladislav Rampášek","Mikhail Galkin","Vijay Prakash Dwivedi","Anh Tuan Luu","Guy Wolf","Dominique Beaini"],"url":"https://arxiv.org/abs/2205.12454","pdfUrl":"https://arxiv.org/pdf/2205.12454v4","codeUrl":"https://github.com/rampasek/GraphGPS","preprint":{"id":"2205.12454","submittedOn":"2022-05-25","url":"https://arxiv.org/abs/2205.12454"},"publication":{"venue":"NeurIPS","year":2022,"kind":"conference","url":"https://proceedings.neurips.cc/paper_files/paper/2022/hash/5d4834a159f1547b267a05a4e2b7cf5e-Abstract-Conference.html"},"citation":"@misc{graphlearninggraphgps2022,\n  title = {{Recipe for a General, Powerful, Scalable Graph Transformer}},\n  author = {Ladislav Rampášek and Mikhail Galkin and Vijay Prakash Dwivedi and Anh Tuan Luu and Guy Wolf and Dominique Beaini},\n  year = {2022},\n  eprint = {2205.12454},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2205.12454}\n}\n","citationPath":"/content/papers/citations/graph-learning/graphgps.053e58fd5309.bib"},"thumbnail":{"src":"/content/papers/assets/e4177a894582643d7811bced6f6c6cf8bb51ff7738da4af5d98115883e1a4bb8.png","width":1116,"height":798},"locale":"en","path":"/content/papers/en/graph-learning/graphgps.88c624c18f31.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"position-and-structure","text":"Position and Structure","depth":3},{"id":"why-message-passing-needs-them","text":"Why Message Passing Needs Them","depth":3},{"id":"message-passing-before-attention-can-lose-information-first","text":"Message Passing before Attention Can Lose Information First","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-ingredients","text":"Three Ingredients","depth":3},{"id":"one-gps-layer","text":"One GPS Layer","depth":3},{"id":"linear-complexity","text":"Linear Complexity","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"ablation-each-paths-contribution","text":"Ablation: Each Path's Contribution","depth":3},{"id":"comparison-with-other-models","text":"Comparison with Other Models","depth":3},{"id":"reassessment-a-year-later","text":"Reassessment a Year Later","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"graph-learning/graph-transformer","relation":"referenced","topic":""},{"id":"graph-learning/san","relation":"referenced","topic":""},{"id":"graph-learning/graphormer","relation":"referenced","topic":""}]}],"text":"To find a molecule's dipole moment, energy gap or zero-point vibrational energy, chemists usually run a density functional theory (DFT) calculation. A small molecule takes roughly a thousand seconds.\n\nWhen drug or materials screening involves millions of candidate molecules, that speed becomes a bottleneck.\n\nJustin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals and George Dahl, from Google Brain, Google and DeepMind, want neural networks to predict these properties directly from molecular graphs, taking around a hundredth of a second per molecule.\n\nPlenty of neural networks could already consume molecular graphs, each with its own name. The paper's first move is to put them all into one framework.\n\n[![Figure 1 of the paper: an organic molecule has two routes to property prediction. The upper route uses DFT, taking around 10 to the power of 3 seconds; the lower route uses a message passing neural network, with atoms exchanging messages step by step, taking around 10 to the power of -2 seconds. Both predict targets such as energies and vibrational frequencies.](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)](/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png)\n\n*Figure 1 from Gilmer et al., taken from [**PDF page 1**](https://arxiv.org/pdf/1704.01212v2#page=1). The original caption is omitted.*\n\n## Defining the Problem\n\nThe authors note that most machine-learning applications in chemistry still depend on feature engineering. Researchers manually design molecular representations—Coulomb matrices, Bag of Bonds, molecular fingerprints—then attach an off-the-shelf regressor.\n\nNeural networks have been tried but are not yet widely adopted. The authors compare this with image modeling before convolutional networks became widespread: evidence is needed that neural networks with suitable inductive biases really work in this domain.\n\nMolecules have natural symmetries. Renumber the atoms and the molecule stays the same; a suitable model should be invariant to graph isomorphism. The authors pursue two goals: identify a common form for existing graph neural networks, then push it to its limits on a practical benchmark.\n\nThat benchmark is QM9: roughly 130,000 small molecules, each with 13 properties computed by DFT. Evaluation uses an intuitive standard, “chemical accuracy,” established in chemistry; errors below that threshold meet the accuracy chemists consider acceptable.\n\n## Solving the Problem\n\n### The Message Passing Framework\n\nMPNNs (Message Passing Neural Networks) divide the forward pass into two phases.\n\nThe **message passing phase** runs for $T$ steps. At each step, node $v$ receives a message from each neighbor $w$, sums them, and updates its state:\n\n$$\nm_v^{t+1}=\\sum_{w\\in N(v)}M_t\\left(h_v^t,h_w^t,e_{vw}\\right)\n$$\n\n$$\nh_v^{t+1}=U_t\\left(h_v^t,m_v^{t+1}\\right)\n$$\n\n$M_t$ is the message function, which sees both sender and receiver states and edge features $e_{vw}$; $U_t$ is the update function.\n\nThe **readout phase** combines the final states of all nodes into a graph representation:\n\n$$\n\\hat y=R\\left(\\left\\{h_v^T\\mid v\\in G\\right\\}\\right)\n$$\n\n$R$ must be invariant to node permutations. All three functions are learnable.\n\nThe authors then show, one by one, how at least eight existing models fit this form: Duvenaud et al.'s molecular-fingerprint convolutions, gated graph neural networks (GG-NNs), Interaction Networks, molecular graph convolutions, deep tensor networks, and three Laplacian-based spectral methods (Bruna et al., ChebNet and GCN).\n\nFor GCN, the message function multiplies the neighboring state by weights and scales it by the two endpoint degrees; the update is ReLU. On a five-node graph, compute it through message and update functions, then compare with GCN's matrix formulation:\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\nn = 5\nedges = [(0, 1), (0, 2), (1, 2), (2, 3), (3, 4)]\nnbrs = {v: [v] for v in range(n)}            # Start with self: a self-loop is an edge too\nfor i, j in edges:\n    nbrs[i].append(j)\n    nbrs[j].append(i)\ndeg = {v: len(nbrs[v]) for v in range(n)}    # Degrees after adding self-loops\nh = rng.normal(size=(n, 3))\nW = rng.normal(size=(3, 3))\n\ndef message(h_v, h_w, v, w):                 # M_t(h_v, h_w, e_vw)\n    return h_w @ W / np.sqrt(deg[v] * deg[w])\n\ndef update(h_v, m_v):                        # U_t(h_v, m_v)\n    return np.maximum(m_v, 0)\n\ndef mpnn_step(h):\n    return np.array([update(h[v], sum(message(h[v], h[w], v, w) for w in nbrs[v]))\n                     for v in range(n)])\n\n# Compare with GCN matrix form ReLU(Â H W)\nA = np.eye(n)\nfor i, j in edges:\n    A[i, j] = A[j, i] = 1\nd = A.sum(1)\nA_hat = A / np.sqrt(np.outer(d, d))\nprint(\"Matches the GCN matrix form:\", np.allclose(mpnn_step(h), np.maximum(A_hat @ h @ W, 0)))\n```\n\n```text\nMatches the GCN matrix form: True\n```\n\nBoth compute the same thing: one describes a single node's perspective, while the other writes the whole graph as a matrix. The framework's benefit is that replacing `message` or `update` gives another model.\n\n### Which Variations Did the Authors Try?\n\nWith a common form in hand, the authors try several choices for each component:\n\n- **Message function**: GG-NN selects a matrix by edge type and multiplies the neighbor's state. The proposed **edge network** maps continuous edge features through a neural network into a $d\\times d$ matrix, allowing quantities such as interatomic distance. Another option, **pair message**, sees both sender and receiver states, but is less stable to train and is not explored further.\n- **Virtual graph elements**: one option adds special “virtual edges” between otherwise unconnected node pairs; another adds a **master node** connected to every node, acting as global scratch space that everyone can read and write at each step. Both let information travel farther during message passing.\n- **Readout function**: beyond GG-NN's sum-based readout, the authors try set2set, designed for sets and expected to be more expressive.\n- **Towers**: split a $d$-dimensional node state into $k$ parts, run message passing separately, then mix them through a shared network. This allows larger hidden states for the same computation.\n\nFor inputs, explicitly representing hydrogen atoms as nodes and including full edge features—bond types and interatomic distances—matter for many targets.\n\n## Discussion\n\n### Main Results on QM9\n\nThe metric is the **error ratio**: mean absolute error divided by the target's chemical-accuracy threshold. A value below 1 achieves chemical accuracy. The authors compare five handcrafted molecular representations, each with an off-the-shelf regressor, two existing MPNN baselines, and their best variant: edge-network messages, set2set readout and explicit hydrogens.\n\n| Method | Mean error ratio over 13 targets |\n| --- | ---: |\n| Molecular fingerprint ECFP4 | 53.97 |\n| Coulomb matrix | 3.37 |\n| Bag of Bonds (BoB) | 2.08 |\n| HDAD | 1.35 |\n| Molecular graph convolution (GC) | 2.59 |\n| GG-NN | 1.36 |\n| **enn-s2s (this paper)** | **0.68** |\n| enn-s2s, ensemble of 5 models | 0.52 |\n\nThe single model sets the best result of the time on all 13 targets, achieving chemical accuracy on 11. Training a separate model for each target also works better than training all 13 together, sometimes by as much as 40%.\n\n### Without Spatial Information\n\nQM9 provides 3D molecular conformations, but real applications often have only connectivity. In a topology-only setting, the authors find that every change allowing distant nodes to exchange information helps:\n\n| Model | Mean error ratio |\n| --- | ---: |\n| GG-NN | 3.47 |\n| GG-NN + virtual edges | 2.90 |\n| GG-NN + master node | 2.62 |\n| GG-NN + set2set readout | 2.57 |\n\nAll three changes improve every one of the 13 targets; set2set achieves chemical accuracy on 5. These experiments add a partial-charge node feature, itself a DFT output unavailable in practical use. The main results above do not use this feature.\n\nTowers bring an unexpected benefit. Intended simply to speed training, they beat the baseline on 12 of 13 targets under both training settings and generalize better.\n\n### Virtual Edges and the Master Node\n\nThese changes occupy only a short passage in the paper, but are interesting in hindsight. More than three years later, Alon and Yahav call GNNs' difficulty transmitting distant information over-squashing; their repair also connects all nodes in the last layer. Later Graph Transformers add a route for all nodes to see one another at every layer.\n\n\n> [!TIP]\n> \n> For over-squashing and the later use of global attention in every layer, see our notes:\n> \n> - **[[20.06] Over-squashing: No Room Left](/blog/papers/graph-learning/over-squashing)**\n> - **[[22.05] GraphGPS: GPS for Graphs](/blog/papers/graph-learning/graphgps)**\n\n\n## Conclusion\n\nThis paper organizes graph neural networks, each previously named separately, into three functions: message, update and readout. The name becomes so widely used that many papers call this whole family message passing.\n\nOn QM9, edge networks, set2set and explicit hydrogens produce the best results of the time on all 13 targets, reaching chemical accuracy on 11.\n\nThe authors urge researchers to push this family to its limits on applications that really matter, finding the implementation details that count, rather than constantly inventing new variants. After reading the paper, those details are often the most useful: whether hydrogens should be nodes, whether to train targets separately, and which readout to use.\n","imageSizes":{"/content/papers/assets/44a847b2362597f5662f2709ea7ecf27df3d1f6c76cd135fde21b0ad814ca891.png":{"width":628,"height":388}}}