{"notes":[{"id":"classic-cnns/highway-networks","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Highway Networks","subtitle":"On the Eve of Residuals","author":"Z. Yuan","paperDate":"2015-05-03","publishedOn":"2025-11-03","world":"external/highway-networks","paperTitle":"Highway Networks","paperUrl":"https://arxiv.org/abs/1505.00387","bibliography":{"title":"Highway Networks","authors":["Rupesh Kumar Srivastava","Klaus Greff","Jürgen Schmidhuber"],"url":"https://arxiv.org/abs/1505.00387","pdfUrl":"https://arxiv.org/pdf/1505.00387v2","codeUrl":null,"preprint":{"id":"1505.00387","submittedOn":"2015-05-03","url":"https://arxiv.org/abs/1505.00387"},"publication":null,"citation":"@misc{classiccnnshighwaynetworks2015,\n  title = {{Highway Networks}},\n  author = {Rupesh Kumar Srivastava and Klaus Greff and Jürgen Schmidhuber},\n  year = {2015},\n  eprint = {1505.00387},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1505.00387}\n}\n","citationPath":"/content/papers/citations/classic-cnns/highway-networks.f7d3ef9547ef.bib"},"thumbnail":{"src":"/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png","width":1344,"height":360},"locale":"en","path":"/content/papers/en/classic-cnns/highway-networks.7580f83fc759.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"adding-two-gates","text":"Adding Two Gates","depth":3},{"id":"lean-toward-pass-straight-through-at-first","text":"Lean Toward “Pass Straight Through” at First","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-deeper-the-easier-to-train","text":"The Deeper, the Easier to Train?","depth":3},{"id":"easy-to-train-but-what-about-accuracy","text":"Easy to Train, but What About Accuracy?","depth":3},{"id":"what-did-the-gates-learn","text":"What Did the Gates Learn?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/resnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/resnet","category":"classic-cnns","legacyPath":"classic-cnns/1512-resnet","legacySlug":"1512-resnet","legacyPaths":[],"title":"ResNet","subtitle":"Convolutional with 100 Layers","author":"Z. Yuan","paperDate":"2015-12-10","publishedOn":"2024-08-08","paperTitle":"Deep Residual Learning for Image Recognition","paperUrl":"https://arxiv.org/abs/1512.03385","bibliography":{"title":"Deep Residual Learning for Image Recognition","authors":["Kaiming He","Xiangyu Zhang","Shaoqing Ren","Jian Sun"],"url":"https://arxiv.org/abs/1512.03385","pdfUrl":"https://arxiv.org/pdf/1512.03385","codeUrl":null,"preprint":{"id":"1512.03385","submittedOn":"2015-12-10","url":"https://arxiv.org/abs/1512.03385"},"publication":{"venue":"CVPR","year":2016,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2016.90"},"citation":"@misc{classiccnnsresnet2015,\n  title = {{Deep Residual Learning for Image Recognition}},\n  author = {Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun},\n  year = {2015},\n  eprint = {1512.03385},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1512.03385}\n}\n","citationPath":"/content/papers/citations/classic-cnns/resnet.cf4014e8baae.bib"},"thumbnail":{"src":"/content/papers/assets/b8fe87fe757249ea6d7bb667fef3196754b9f80e28bf842a69b540ca94585d08.jpg","width":1224,"height":408},"locale":"en","path":"/content/papers/en/classic-cnns/resnet.b65375cb9846.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-is-residual","text":"What is Residual?","depth":3},{"id":"residual-learning","text":"Residual Learning","depth":3},{"id":"residual-module","text":"Residual Module","depth":3},{"id":"network-architecture","text":"Network Architecture","depth":3},{"id":"architectures-and-computation","text":"Architectures and Computation","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"resnet-18-vs-resnet-34","text":"ResNet-18 vs ResNet-34","depth":3},{"id":"deeper-bottleneck-architectures","text":"Deeper Bottleneck Architectures","depth":3},{"id":"performance-on-imagenet","text":"Performance on ImageNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/vgg","relation":"background","topic":"Training deeper convolutional networks"},{"id":"classic-cnns/densenet","relation":"next","topic":"Training deeper convolutional networks"}]}],"text":"By 2015, the benefits of depth were well understood. In the introduction, the authors cite ImageNet as an example: with deeper networks and smaller receptive fields, top-5 accuracy climbed within a few years from about 84% with AlexNet to about 95%.\n\nThe trouble is that once a network gets deep, it becomes hard to train.\n\nTo get deep networks to train, people at the time came up with several approaches: initializations designed to keep the signal variance stable; training a shallow network first and then deepening it one stage at a time; or attaching auxiliary loss functions to intermediate layers. All of these help, but they're all ways of working around depth.\n\nIn May 2015, Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber of IDSIA in Switzerland released this short paper, which comes at the problem from another angle: instead of trying to get the signal safely through every layer, give it a path that can skip the layer.\n\n\n> [!NOTE]\n> \n> This paper is an extended abstract for the ICML 2015 Deep Learning Workshop, only a few pages long. The same authors later expanded the work into [Training Very Deep Networks](https://arxiv.org/abs/1507.06228), published at NIPS 2015 with more experiments and analysis; this note covers the earlier short version.\n\n\n## Defining the Problem\n\nIn an ordinary feedforward network, each layer applies a nonlinear transformation to its input:\n\n$$\ny = H(x, W_H)\n$$\n\n$H$ is usually an affine transformation followed by an activation function. When the network is very deep, training with SGD often stalls right from the start, unless the weights are initialized just right so that the signal neither grows nor decays during forward and backward propagation (Glorot initialization or He initialization, for example).\n\nInitializations like these have two limitations. First, they depend on the exact form of $H$: switch to another activation function, and you may have to derive a new initialization. Second, the authors found that even with He initialization, very deep plain networks are still hard to optimize, as we'll see in the experiments.\n\nSo is there an architecture that can be trained directly with SGD, whatever $H$ looks like and however many layers are stacked?\n\n## Solving the Problem\n\n### Adding Two Gates\n\nThe authors' inspiration comes from LSTMs. An LSTM uses gates to decide how much of its memory to keep and how much to update; here the same idea is carried over to every layer of a feedforward network.\n\nThey define two additional nonlinear transformations, the transform gate $T(x, W_T)$ and the carry gate $C(x, W_C)$:\n\n$$\ny = H(x, W_H)\\cdot T(x, W_T) + x\\cdot C(x, W_C)\n$$\n\n$T$ decides how much of the output comes from “transforming the input,” and $C$ decides how much comes from “carrying the input straight through.” For simplicity, the paper sets $C = 1 - T$:\n\n$$\ny = H(x, W_H)\\cdot T(x, W_T) + x\\cdot\\big(1 - T(x, W_T)\\big)\n$$\n\nLook at the two extremes:\n\n$$\ny=\\begin{cases}x, & T=0\\\\ H(x, W_H), & T=1\\end{cases}\n\\qquad\\qquad\n\\frac{dy}{dx}=\\begin{cases}I, & T=0\\\\ H'(x, W_H), & T=1\\end{cases}\n$$\n\nWhen $T=0$, the layer simply hands over its input; the derivative with respect to the input is the identity matrix, and gradients flow back untouched. When $T=1$, it's an ordinary network layer. In practice $T$ lies somewhere in between, so each layer can adjust smoothly between “ordinary layer” and “pass straight through.”\n\n$T$ here is a vector of the same length as $x$, so each unit has its own gate: the output of the $i$-th unit is $y_i = H_i(x)\\,T_i(x) + x_i\\,\\big(1 - T_i(x)\\big)$. That's why $x$, $y$, $H(x)$, and $T(x)$ must all have the same dimensionality.\n\nWhat if you need to change the dimensionality? The authors offer two options: subsample or zero-pad $x$ as appropriate, or use an ordinary layer to change the dimensionality and follow it with highway layers. The paper uses the latter.\n\nThe convolutional version of a highway layer works the same way: $H$ and $T$ are both computed with convolutions, each with its own shared weights, and zero padding keeps their feature maps the same size as the input.\n\n### Lean Toward “Pass Straight Through” at First\n\nThe transform gate has the form:\n\n$$\nT(x) = \\sigma(W_T^T x + b_T)\n$$\n\nThe authors give an initialization that doesn't depend on $H$: set the bias $b_T$ to a negative value, such as $-1$ or $-3$, so that the network starts out biased toward carrying, that is, toward passing straight through. This idea also comes from LSTMs: in 1999, Gers et al. suggested initially biasing the gates of an LSTM in a particular direction so that the model could bridge long-range dependencies early in learning.\n\nWith a negative bias, how small does $T$ get?\n\n```python\nimport torch\n\nfor b_T in (0.0, -1.0, -3.0):\n    print(f\"b_T = {b_T:4}: T is about {torch.sigmoid(torch.tensor(b_T)).item():.3f}\")\n```\n\n```text\nb_T =  0.0: T is about 0.500\nb_T = -1.0: T is about 0.269\nb_T = -3.0: T is about 0.047\n```\n\nWith $b_T=-3$, only about 5% of each layer's output comes from the transformation, and 95% is the input carried straight through. Because a sigmoid's output always lies strictly between 0 and 1, the two extreme conditions above never hold exactly; they only come very close.\n\nWhat about after stacking many layers? Using 50-dimensional vectors, let's compare a 50-layer plain network with a 50-layer highway network right after initialization and see how much the output still resembles the input:\n\n```python\nimport torch\nimport torch.nn.functional as F\n\ntorch.manual_seed(0)\nD, depth = 50, 50\nx = torch.randn(D)\n\ndef run(highway, b_T=-3.0):\n    h = x\n    for _ in range(depth):\n        W_H = torch.randn(D, D) * (2 / D) ** 0.5     # He initialization\n        H = F.relu(W_H @ h)\n        if highway:\n            W_T = torch.randn(D, D) * (1 / D) ** 0.5\n            T = torch.sigmoid(W_T @ h + b_T)\n            h = H * T + h * (1 - T)\n        else:\n            h = H\n    return F.cosine_similarity(h, x, dim=0).item()\n\nprint(f\"Plain   after {depth} layers, cosine similarity between output and input: {run(False):.2f}\")\nprint(f\"Highway after {depth} layers, cosine similarity between output and input: {run(True):.2f}\")\n```\n\n```text\nPlain   after 50 layers, cosine similarity between output and input: 0.01\nHighway after 50 layers, cosine similarity between output and input: 0.44\n```\n\nAfter 50 layers, the plain network's output has almost nothing to do with its input; the highway network still keeps a good part of the input's direction. Across a few other random seeds, the plain network lands between $-0.13$ and $0.20$, and the highway network between $0.26$ and $0.51$.\n\nThis is only an illustration at initialization, not an experiment from the paper. The point it makes: from the very start of training, information and gradients have a path that goes through almost no transformation, and from there the network can gradually learn when to transform and by how much. In their experiments, the authors found that as long as the bias is negative, very deep networks can start learning, whichever zero-mean initial distribution is used for $W_H$ and whichever activation function $H$ uses.\n\n## Discussion\n\n### The Deeper, the Easier to Train?\n\nTo look at optimization alone, without mixing in generalization, the authors measure cross-entropy on the MNIST **training set**.\n\nEach network's first layer is always an ordinary fully connected layer, followed by 9, 19, 49, or 99 more layers and finally a softmax. Plain networks have 71 units per layer and highway networks 50, so the two have roughly the same number of parameters. For fairness, each kind of network gets a random search of 40 runs over the learning rate, momentum, learning-rate decay, and the activation function of $H$ (ReLU or tanh); for highway networks, the search also covers the transform gate bias ($-1$ to $-10$). All other weights use He initialization.\n\n[![Four training-curve plots side by side, titled depth 10, 20, 50, and 100. The vertical axis is mean cross-entropy (log scale), and the horizontal axis is epochs. Blue lines are plain networks and green lines are highway networks. At depth 10 the blue line ends lowest; as depth increases the blue line ends higher and higher, and at depth 100 it barely comes down; the green line drops very low in all four plots](/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png)](/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png)\n\n*Figure 1 from the paper, excerpted from [**page 4 of the v2 PDF**](https://arxiv.org/pdf/1505.00387v2#page=4), CC BY 4.0. Each curve is the best result of the random search for that setting; blue lines are plain networks, green lines are highway networks.*\n\nThe vertical axis is logarithmic: each gridline down is ten times smaller.\n\nAt depth 10, the plain network does very well and ends up even lower than the highway network. As depth increases, the plain network has a harder and harder time coming down; at depth 100, the blue line stays stuck up high.\n\nThe green line, on the other hand, shows no visible effect of depth. The authors point out that the 100-layer highway network's final error is about an order of magnitude better than the 10-layer highway network's, on par with the 10-layer plain network; and highway networks consistently converge faster than plain ones. They also started training a 900-layer highway network on CIFAR-100. At the time of writing it had only reached epoch 80, but so far it had shown no signs of optimization difficulties.\n\n### Easy to Train, but What About Accuracy?\n\nBeing easy to train is one thing; performance on the test set is another. The authors compare against FitNets, the thin, deep networks built with maxout that Romero et al. proposed in 2014. Romero et al. found that with parameters limited to about 250,000 and multiplications to about 30 million, plain backpropagation could train only up to about 5 layers; deeper networks required first training a shallow teacher network, then training the student in two stages, using the teacher's intermediate layers as hints together with the teacher's soft targets.\n\nThe highway networks, by contrast, are all trained directly with backpropagation. The comparison is on CIFAR-10 augmented with random translations:\n\n| Network | Layers | Parameters | Test accuracy |\n| --- | ---: | ---: | ---: |\n| Teacher (the FitNets teacher) | 5 | ~9M | 90.18% |\n| Fitnet 1 | 11 | ~250K | 89.01% |\n| Fitnet 4 | 19 | ~2.5M | 91.61% |\n| Highway 1 (counterpart of Fitnet 1) | 11 | ~236K | 89.18% |\n| Highway 2 (counterpart of Fitnet 4) | 19 | ~2.3M | **92.24%** |\n| Highway 3* | 19 | ~1.4M | 90.68% |\n| Highway 4* | 32 | ~1.25M | 90.34% |\n\n*Table: Excerpted from Table 1 of the paper. Networks marked * were trained on only 40,000 of the 50,000 training images. The paper's text calls the counterpart of Fitnet 4 “Highway 4,” while the table labels it Highway 2; we follow the table here.*\n\nAt similar parameter counts, the directly trained highway networks match or beat the accuracy of the two-stage FitNets. The thinner, deeper Highway 3 and 4, with only a little over a million parameters, also perform about as well as the 9-million-parameter teacher network.\n\n### What Did the Gates Learn?\n\nFinally, the authors open up trained 50-layer fully connected highway networks to see how the gates actually behave:\n\n[![Heat maps in two rows and four columns. The top row is MNIST and the bottom row is CIFAR-100; in each plot the vertical axis is depth and the horizontal axis is the 50 units. The first column is the transform gate biases; the second is the mean transform gate output over 10,000 samples; the third is the transform gate output for a single sample, mostly low blue values with only scattered bright spots; the fourth is the block output for a single sample, showing vertical stripes running from top to bottom](/content/papers/assets/6457aa77c955b4147a7edaeae2028f69f8bf7daf4a7ad70d6a53384f6ed9bfff.png)](/content/papers/assets/6457aa77c955b4147a7edaeae2028f69f8bf7daf4a7ad70d6a53384f6ed9bfff.png)\n\n*Figure 2 from the paper, excerpted from [**page 5 of the v2 PDF**](https://arxiv.org/pdf/1505.00387v2#page=5), CC BY 4.0. Top row MNIST, bottom row CIFAR-100. The four columns are, in order, the transform gate biases, the mean gate output over 10,000 samples, the gate output for a single sample, and the block output for that same sample.*\n\nA few things in this figure are worth reading side by side.\n\n**The biases are more negative than expected.** The two networks' biases were initialized to $-2$ and $-4$, respectively. The authors expected training to push them up, but most actually became even more negative. On CIFAR-100, the biases rise with depth, forming a gradient, and this gradient runs exactly opposite to the mean gate output in the second column. The authors' reading: the strongly negative biases in the shallow layers make the gates more selective, so they open only for particular inputs. The very sparse single-sample gate output in the third column also supports this reading.\n\n**Most units pass straight through.** The block outputs in the fourth column form vertical stripes: many units' outputs stay almost unchanged across dozens of layers. These are the so-called information highways. Most of the change in output happens in the early layers, about the first 10 for MNIST and about the first 30 for CIFAR-100; the authors suspect this has to do with CIFAR-100 being more complex.\n\nFrom this, the authors conclude that the gates are more than a tool for making training easier: trained networks really do use them to decide which route information takes, and the decisions vary strongly with the input.\n\n## Conclusion\n\nHighway Networks let each layer learn “how much to transform and how much to pass straight through.” Paired with an initialization biased toward passing straight through, even 100-layer networks can be trained with simple SGD; on CIFAR-10, they reach the level of FitNets without needing a teacher network.\n\nSeven months later, Microsoft Research's ResNet used ungated identity shortcuts to push networks to 152 layers and won ILSVRC 2015. The ResNet paper mentions highway networks in its related work and spells out the differences: highway gates depend on the input and have parameters, and when a gate is closed, the layer no longer learns a residual; ResNet's identity shortcuts have no parameters and are never closed, so all information always passes through. Highway networks had also not yet shown accuracy gains at extreme depths (over 100 layers, for example).\n\n\n> [!TIP]\n> \n> For ResNet's approach, see our note:\n> \n> - **[[15.12] ResNet: Convolutional with 100 Layers](/blog/papers/classic-cnns/resnet)**\n\n\nThis relationship was later described differently, too. In his 2025 technical report [Who invented deep residual learning?](https://arxiv.org/abs/2509.24732), Schmidhuber describes ResNet as “like an open-gated Highway Net” and traces the origins of residual connections back to his lab's work since 1991.\n\nIf you're training a very deep network today, you probably won't pick Highway: ResNet's identity shortcuts have no gates and no extra parameters, and you don't have to worry about what to initialize a gate bias to.\n\nStill, “give the signal a path without transformation” is something Highway made work first, in hundred-layer networks. Next, you can see how ResNet tore that gate down.\n","imageSizes":{"/content/papers/assets/2cef37bd2a2388f57a76965431810cb86f976db701ba1647b75c81c9a59ac779.png":{"width":1344,"height":360},"/content/papers/assets/6457aa77c955b4147a7edaeae2028f69f8bf7daf4a7ad70d6a53384f6ed9bfff.png":{"width":1352,"height":724}}}