{"notes":[{"id":"classic-cnns/regnet","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RegNet","subtitle":"Do Good Architectures All Look Alike?","author":"Z. Yuan","paperDate":"2020-03-30","publishedOn":"2024-12-16","world":"external/regnet","paperTitle":"Designing Network Design Spaces","paperUrl":"https://arxiv.org/abs/2003.13678","bibliography":{"title":"Designing Network Design Spaces","authors":["Ilija Radosavovic","Raj Prateek Kosaraju","Ross Girshick","Kaiming He","Piotr Dollár"],"url":"https://arxiv.org/abs/2003.13678","pdfUrl":"https://arxiv.org/pdf/2003.13678v1","codeUrl":null,"preprint":{"id":"2003.13678","submittedOn":"2020-03-30","url":"https://arxiv.org/abs/2003.13678"},"publication":{"venue":"CVPR","year":2020,"kind":"conference","url":"https://openaccess.thecvf.com/content_CVPR_2020/html/Radosavovic_Designing_Network_Design_Spaces_CVPR_2020_paper.html"},"citation":"@misc{classiccnnsregnet2020,\n  title = {{Designing Network Design Spaces}},\n  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Dollár},\n  year = {2020},\n  eprint = {2003.13678},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2003.13678}\n}\n","citationPath":"/content/papers/citations/classic-cnns/regnet.51d4a967c43d.bib"},"thumbnail":{"src":"/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png","width":1013,"height":550},"locale":"en","path":"/content/papers/en/classic-cnns/regnet.5bb32cbae384.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"what-manual-design-and-nas-each-lack","text":"What Manual Design and NAS Each Lack","depth":3},{"id":"how-to-compare-two-design-spaces","text":"How to Compare Two Design Spaces","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-starting-point-anynet","text":"The Starting Point: AnyNet","depth":3},{"id":"narrowing-it-down-step-by-step","text":"Narrowing It Down Step by Step","depth":3},{"id":"width-is-a-straight-line","text":"Width Is a Straight Line","depth":3},{"id":"regnet","text":"RegNet","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"where-it-breaks-with-common-practice","text":"Where It Breaks with Common Practice","depth":3},{"id":"compared-with-resnet-and-resnext","text":"Compared with ResNet and ResNeXt","depth":3},{"id":"the-mobile-regime","text":"The Mobile Regime","depth":3},{"id":"compared-with-efficientnet","text":"Compared with EfficientNet","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/nasnet","relation":"referenced","topic":""},{"id":"classic-cnns/efficientnet","relation":"referenced","topic":""}]},{"id":"classic-cnns/nasnet","category":"classic-cnns","legacyPath":"classic-cnns/1707-nasnet","legacySlug":"1707-nasnet","legacyPaths":[],"title":"NASNet","subtitle":"Searching for Network Architectures","author":"Z. Yuan","paperDate":"2017-07-21","publishedOn":"2024-06-26","paperTitle":"Learning Transferable Architectures for Scalable Image Recognition","paperUrl":"https://arxiv.org/abs/1707.07012","bibliography":{"title":"Learning Transferable Architectures for Scalable Image Recognition","authors":["Barret Zoph","Vijay Vasudevan","Jonathon Shlens","Quoc V. Le"],"url":"https://arxiv.org/abs/1707.07012","pdfUrl":"https://arxiv.org/pdf/1707.07012","codeUrl":null,"preprint":{"id":"1707.07012","submittedOn":"2017-07-21","url":"https://arxiv.org/abs/1707.07012"},"publication":{"venue":"CVPR","year":2018,"kind":"conference","url":"https://doi.org/10.1109/cvpr.2018.00907"},"citation":"@misc{classiccnnsnasnet2017,\n  title = {{Learning Transferable Architectures for Scalable Image Recognition}},\n  author = {Barret Zoph and Vijay Vasudevan and Jonathon Shlens and Quoc V. Le},\n  year = {2017},\n  eprint = {1707.07012},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1707.07012}\n}\n","citationPath":"/content/papers/citations/classic-cnns/nasnet.f6a561f13f90.bib"},"thumbnail":{"src":"/content/papers/assets/b2d2e9edd0aa83df25d43fcba9a942774de024dfa090f0771473452abb1ff417.jpg","width":930,"height":966},"locale":"en","path":"/content/papers/en/classic-cnns/nasnet.2e5b9ff03912.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"transferable-architectures","text":"Transferable Architectures","depth":3},{"id":"fundamental-structures","text":"Fundamental Structures","depth":3},{"id":"the-search-process","text":"The Search Process","depth":3},{"id":"droppath-module","text":"DropPath Module","depth":3},{"id":"search-experiment-results","text":"Search Experiment Results","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"results-on-cifar-10","text":"Results on CIFAR-10","depth":3},{"id":"transferring-architectures-to-imagenet","text":"Transferring Architectures to ImageNet","depth":3},{"id":"evaluation-on-mobile-devices","text":"Evaluation on Mobile Devices","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/efficientnet","relation":"next","topic":"Architecture search and efficient scaling"}]},{"id":"classic-cnns/efficientnet","category":"classic-cnns","legacyPath":"classic-cnns/1905-efficientnet","legacySlug":"1905-efficientnet","legacyPaths":[],"title":"EfficientNet","subtitle":"Compound Scaling","author":"Z. Yuan","paperDate":"2019-05-28","publishedOn":"2024-07-04","paperTitle":"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks","paperUrl":"https://arxiv.org/abs/1905.11946","bibliography":{"title":"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks","authors":["Mingxing Tan","Quoc V. Le"],"url":"https://arxiv.org/abs/1905.11946","pdfUrl":"https://arxiv.org/pdf/1905.11946","codeUrl":null,"preprint":{"id":"1905.11946","submittedOn":"2019-05-28","url":"https://arxiv.org/abs/1905.11946"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1905.11946"},"citation":"@misc{classiccnnsefficientnet2019,\n  title = {{EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}},\n  author = {Mingxing Tan and Quoc V. Le},\n  year = {2019},\n  eprint = {1905.11946},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1905.11946}\n}\n","citationPath":"/content/papers/citations/classic-cnns/efficientnet.ad7d6744b5d4.bib"},"thumbnail":{"src":"/content/papers/assets/f8202301fddceafac70e33ea34ffeb6b2255d2617c880baee08c51ce4c8a4c93.jpg","width":1596,"height":674},"locale":"en","path":"/content/papers/en/classic-cnns/efficientnet.0edcea10a122.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-key-observations","text":"Two Key Observations","depth":3},{"id":"compound-scaling","text":"Compound Scaling","depth":3},{"id":"base-model-architecture","text":"Base Model Architecture","depth":3},{"id":"model-scaling-architecture","text":"Model Scaling Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"importance-of-compound-scaling","text":"Importance of Compound Scaling","depth":3},{"id":"results-on-imagenet","text":"Results on ImageNet","depth":3},{"id":"inference-speed","text":"Inference Speed","depth":3},{"id":"parameter-accuracy-curve","text":"Parameter-Accuracy Curve","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"compound-scaling-1":"compound-scaling"},"guide":null,"relatedReading":[{"id":"classic-cnns/nasnet","relation":"background","topic":"Architecture search and efficient scaling"},{"id":"classic-cnns/efficientnet-v2","relation":"next","topic":"Architecture search and efficient scaling"}]}],"text":"Around 2019, one of the hottest ways to design convolutional networks was neural architecture search (NAS). You define a search space and let an algorithm find the best network within it; that's how NASNet and EfficientNet came about.\n\n\n> [!TIP]\n> \n> For these two, you can start with our notes:\n> \n> - **[[17.07] NASNet: Searching for Network Architectures](/blog/papers/classic-cnns/nasnet)**\n> - **[[19.05] EfficientNet: Compound Scaling](/blog/papers/classic-cnns/efficientnet)**\n\n\nThe networks found this way are strong, but what we get is a single network, a set of numbers tuned for one level of compute and one kind of hardware. Why does it look like this? How should it change for a different level of compute? The search result itself can't answer that.\n\nIlija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár of Facebook AI Research wanted to take a different approach. Instead of designing a single network, they design “network design spaces,” finding design principles in how a large population of networks performs.\n\n## Defining the Problem\n\n### What Manual Design and NAS Each Lack\n\nLook back at LeNet, AlexNet, VGG, and ResNet. These hand-designed networks left behind more than models; they also left principles such as using convolution, scaling up data and networks, going deeper, and adding residuals. These principles carry over to many new settings. The trouble is that once there are many design choices, it gets very hard to find the best combination by trying them one by one by hand.\n\nNAS solves the “trying” problem, but what it produces is a network instance tuned for a specific setting. What the authors want are networks that are simple, easy to understand, and generalize to different settings, along with the principles behind them.\n\nAs for different compute requirements, the common practice is to take one network and scale its depth, width, or resolution by some rule, with EfficientNet's compound scaling as one example. But that still starts from a single network.\n\n### How to Compare Two Design Spaces\n\nIf what you're designing is “a population of networks,” you need a way to compare how good one population is against another. The authors use a tool that Radosavovic et al. proposed the year before, which randomly samples $n$ models from a design space, trains them all, and looks at the distribution of their errors.\n\nConcretely, the tool is the error empirical distribution function (EDF):\n\n$$\nF(e)=\\frac{1}{n}\\sum_{i=1}^{n}\\mathbf{1}[e_i<e].\n$$\n\n$F(e)$ is simply “the fraction of models with error below $e$.” The closer the curve is to the upper left, the more good models the space contains.\n\n[![On the left, three nested ellipses: the outermost blue one is A, the middle orange one B, and the innermost yellow one C, with arrows pointing from A to B and then to C. On the right, three cumulative probability curves, with error (40 to 60) on the horizontal axis and cumulative probability (0 to 1) on the vertical axis. Blue A is the flattest and furthest right; orange B shifts to the left; yellow C is furthest left and steepest](/content/papers/assets/526c9907c4f475ec129b14c60e3189345920cb53e55b91dea16ee61a5c64c0b2.png)](/content/papers/assets/526c9907c4f475ec129b14c60e3189345920cb53e55b91dea16ee61a5c64c0b2.png)\n\n*Figure 1 from the paper, excerpted from [**page 1 of the v1 PDF**](https://arxiv.org/pdf/2003.13678v1#page=1). Design space design: each step shrinks space A into a simpler or better B, then C; on the right are the error EDFs of all three.*\n\nThe left panel shows the authors' process. Starting from a very large space A, each step adds one constraint, giving a smaller B and then C. The right panel shows the EDF obtained by sampling and training models from each of the three spaces; C's curve is furthest left, meaning its models are the best overall.\n\nComparing whole distributions is more robust than comparing only the best model found in each of two spaces. The authors also hope that principles holding for a whole population of models stand a better chance of generalizing to other settings.\n\nSampling and training hundreds of models is expensive. The authors do their design in a “low-compute, short-training” regime, with each model at about 400 million multiply-adds (400MF), trained for only 10 epochs on ImageNet. At that scale, training 100 models takes about as much compute as training one ResNet-50 for 100 epochs.\n\n## Solving the Problem\n\n### The Starting Point: AnyNet\n\nThe authors begin with a very unconstrained design space called AnyNet. A network has three parts. The stem is a single stride-2 $3\\times3$ convolution with 32 channels; the head is average pooling plus a fully connected layer; the body in the middle is where the action is, with 4 stages that halve the resolution one after another, each made of several identical blocks.\n\nThe block is a ResNeXt-style residual bottleneck with grouped convolution, which the authors call the X block:\n\n[![Diagram of the X block. (a) The stride-1 version: input wi, ri, ri passes through a 1x1 s=1 convolution to become wi/bi channels, then a 3x3 grouped convolution with group width gi, s=1, then a 1x1 s=1 convolution back to wi channels, and is finally added to the shortcut on the right. (b) The stride-2 version: the middle 3x3 convolution has stride 2, and the shortcut gains a 1x1 s=2 convolution](/content/papers/assets/a529e6e4b1fdd3d828f2fabfc86390e33c5501faa9cca45a206d01dc23706bf0.png)](/content/papers/assets/a529e6e4b1fdd3d828f2fabfc86390e33c5501faa9cca45a206d01dc23706bf0.png)\n\n*Figure 4 from the paper, excerpted from [**page 4 of the v1 PDF**](https://arxiv.org/pdf/2003.13678v1#page=4). The X block: a $1\\times1$ convolution, a $3\\times3$ grouped convolution, and a $1\\times1$ convolution, each followed by BN and ReLU.*\n\nEach stage $i$ has four tunable parameters:\n\n- **Number of blocks $d_i$**\n- **Block width $w_i$**: the number of output channels\n- **Bottleneck ratio $b_i$**: the middle layer has $w_i/b_i$ channels\n- **Group width $g_i$**: the number of channels in each group of the $3\\times3$ grouped convolution\n\nFour stages with four parameters each make 16 degrees of freedom. The sampling ranges are $d_i\\le16$, $w_i\\le1024$ and divisible by 8, $b_i\\in\\{1,2,4\\}$, and $g_i\\in\\{1,2,\\ldots,32\\}$, which together give about $10^{18}$ possible networks. The input is fixed at $224\\times224$.\n\nRather than searching these $10^{18}$ networks for the single best one, the authors add constraints step by step, and each step has to make the space simpler and easier to understand without reducing the number of good models.\n\n### Narrowing It Down Step by Step\n\nEach time they add a constraint, the authors sample and train 500 new models to see whether the EDF gets worse:\n\n| Space | New constraint | Degrees of freedom | Combinations | Min / mean error |\n| --- | --- | ---: | ---: | ---: |\n| AnyNetX$_A$ | None | 16 | $\\sim1.8\\cdot10^{18}$ | 39.0 / 49.0 |\n| AnyNetX$_B$ | All stages share $b$ | 13 | $\\sim6.8\\cdot10^{16}$ | 39.0 / 49.2 |\n| AnyNetX$_C$ | All stages share $g$ | 10 | $\\sim3.2\\cdot10^{14}$ | 38.9 / 49.4 |\n| AnyNetX$_D$ | $w_{i+1}\\ge w_i$ | 10 | $\\sim1.3\\cdot10^{13}$ | 38.7 / 43.2 |\n| AnyNetX$_E$ | $d_{i+1}\\ge d_i$ | 10 | $\\sim5.5\\cdot10^{11}$ | 38.7 / 42.7 |\n| RegNetX | Quantized linear | 6 | $\\sim3.0\\cdot10^{8}$ | 38.2 / 41.0 |\n\n*Table: Combination counts compiled from Table 1 of the paper; error rates from the legends of Figures 5, 7, and 9. ImageNet top-1 error (%) at 400MF with 10 epochs of training.*\n\nThe first two steps make all stages share the same $b$ and $g$, removing 6 degrees of freedom while leaving the EDF almost unchanged, which shows that these two parameters don't need to be tuned separately for each stage.\n\nThe third step comes from observation. The authors plotted a few good and a few bad networks from AnyNetX$_C$ and found that the good networks all get wider toward the end. Adding the constraint $w_{i+1}\\ge w_i$ drops the mean error from 49.4% to 43.2% in one go, and the best model doesn't get any worse. The fourth step, $d_{i+1}\\ge d_i$, improves the results a little more.\n\nAt this point, the space has shrunk by a factor of more than three million, and the good models are more concentrated than ever.\n\n### Width Is a Straight Line\n\nNext, the authors plot the 20 best models from AnyNetX$_E$ on a single chart, with the block index on the horizontal axis and the block width on the vertical axis:\n\n[![Top left: 20 gray staircase curves showing the per-block widths of the 20 best models (log-scale vertical axis), with a smooth black curve running through the middle of them. Top middle and right: each shows one good model's actual widths (solid blue staircase), a linear fit (dotted line), and the quantized fit (dashed staircase), with the two staircases nearly overlapping. Bottom row, three scatter plots: fitting error efit versus error for every model in AnyNetXC, AnyNetXD, and AnyNetXE; the good models all cluster where efit is near 0, and from C to E the points gather further and further to the left](/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png)](/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png)\n\n*Figure 8 from the paper, excerpted from [**page 5 of the v1 PDF**](https://arxiv.org/pdf/2003.13678v1#page=5). The widths of good models can be described by a quantized linear function; models with a large fitting error $e_\\text{fit}$ also tend to have higher error.*\n\nIn the top-left chart, every model's gray staircase is different, but the overall trend can be described by a very simple straight line, $w_j=48\\cdot(j+1)$, which is the black curve (the vertical axis is logarithmic, so it looks bent).\n\nThe line gives every block a different width, but in a real network, all the blocks in a stage share the same width, forming a staircase. The authors turn the line into a staircase with three equations.\n\nFirst, a straight line gives each block a continuous width:\n\n$$\nu_j=w_0+w_a\\cdot j,\\qquad 0\\le j<d.\n$$\n\n$w_0$ is the initial width, $w_a$ is the slope, and $d$ is the total number of blocks. Next, a parameter $w_m$ is introduced, and $u_j$ is written as $w_0$ times some power of $w_m$:\n\n$$\nu_j=w_0\\cdot w_m^{s_j}.\n$$\n\nFinally, the exponent $s_j$ is rounded:\n\n$$\nw_j=w_0\\cdot w_m^{\\lfloor s_j\\rceil}.\n$$\n\nOnce the exponents are integers, the widths can only take a few values, $w_0$, $w_0w_m$, $w_0w_m^2$, and so on; the blocks with the same width form a stage, so $w_m$ is the width multiplier between adjacent stages.\n\nLet's actually compute it for RegNetX-1.6GF. Its parameters in torchvision are $d=18$, $w_0=80$, $w_a=34.01$, $w_m=2.25$, and group width $g=24$:\n\n```python\nimport math\n\nd, w0, wa, wm, g = 18, 80, 34.01, 2.25, 24        # RegNetX-1.6GF\n\nu = [w0 + wa * j for j in range(d)]                 # Eq. (2): a continuous width for each block\ns = [math.log(uj / w0) / math.log(wm) for uj in u]  # Eq. (3): convert to the exponent in w0 · wm^s\nw = [w0 * wm ** round(sj) for sj in s]              # Eq. (4): round the exponent, turning the widths into a staircase\nw = [round(wj / 8) * 8 for wj in w]                 # round widths to multiples of 8\n\nfor j in (0, 1, 2, 5, 6, 15, 16, 17):\n    print(f\"Block {j:>2}: u = {u[j]:6.1f}, s = {s[j]:4.2f} → rounded {round(s[j])}, width {w[j]}\")\n\nstages = sorted(set(w))\ndepths = [w.count(x) for x in stages]\nwidths = [round(x / g) * g for x in stages]         # then adjust to multiples of the group width\nprint(\"Blocks per stage:\", depths)\nprint(\"Width per stage:\", widths)\n```\n\n```text\nBlock  0: u =   80.0, s = 0.00 → rounded 0, width 80\nBlock  1: u =  114.0, s = 0.44 → rounded 0, width 80\nBlock  2: u =  148.0, s = 0.76 → rounded 1, width 176\nBlock  5: u =  250.0, s = 1.41 → rounded 1, width 176\nBlock  6: u =  284.1, s = 1.56 → rounded 2, width 408\nBlock 15: u =  590.1, s = 2.46 → rounded 2, width 408\nBlock 16: u =  624.2, s = 2.53 → rounded 3, width 912\nBlock 17: u =  658.2, s = 2.60 → rounded 3, width 912\nBlocks per stage: [2, 4, 10, 2]\nWidth per stage: [72, 168, 408, 912]\n```\n\nThe widths of the 18 blocks climb along the line, and once the exponents are rounded, they split into four segments, giving 2, 4, 10, and 2 blocks per stage, with widths of 72, 168, 408, and 912. This matches RegNetX-1.6GF as listed in the paper, and it also matches the model torchvision builds. The number of blocks per stage doesn't have to be specified separately; the line and the rounding determine it. torchvision's `regnet_x_*` and `regnet_y_*`, as well as timm's `regnetx_*` and `regnety_*`, are all generated with these formulas, so just pick one by compute.\n\nThe authors also went back to check this description. They fit the quantized line to every model from AnyNetX$_C$ through AnyNetX$_E$ and computed the fitting error $e_\\text{fit}$. The scatter plots in the bottom row of Figure 8 show that the best models in each space all have fitting errors near 0, and that from C to E, the models as a whole conform more and more closely to this structure.\n\n### RegNet\n\nSince the good models all look like this, the authors simply keep only networks with this structure. A network is then determined by four parameters, $d$, $w_0$, $w_a$, and $w_m$, plus $b$ and $g$. The sampling ranges are $d<64$, $w_0,w_a<256$, and $1.5\\le w_m\\le3$. The authors call this space RegNet, meaning it contains only regular networks.\n\nThe degrees of freedom drop from 16 to 6 and the size of the space shrinks by nearly 10 orders of magnitude, yet the mean error falls from 49.0% to 41.0%, and the best model gets slightly better too. Good models are concentrated enough that randomly sampling about 32 models is likely to turn up a good one.\n\nThe authors also confirmed that these conclusions hold beyond the low-compute, short-training setting. With higher compute, longer training, 5-stage networks, or other block types, the three spaces always rank in the same order, RegNetX first, then AnyNetX$_E$, then AnyNetX$_A$.\n\n## Discussion\n\n### Where It Breaks with Common Practice\n\nSince good models are concentrated in RegNetX, the authors switch to sampling 100 models at each compute level and training them for 25 epochs, to look at the trend of each parameter:\n\n[![Six line charts, each with compute (0.2 to 12.8 GFLOPs) on the horizontal axis, a pale all range, a blue good range, and a black best line. d: the best depth stays roughly between 16 and 23, mostly 20; b: the best value is always 1; wm: the best value is roughly between 2.2 and 2.8; g, wa, w0: the best values all grow as compute increases](/content/papers/assets/88e61c9524447ce29968da46650f265df2c5f591a45aaf965adef78e49826d51.png)](/content/papers/assets/88e61c9524447ce29968da46650f265df2c5f591a45aaf965adef78e49826d51.png)\n\n*Figure 11 from the paper, excerpted from [**page 7 of the v1 PDF**](https://arxiv.org/pdf/2003.13678v1#page=7). Trends of each RegNetX parameter across compute levels; the blue region is the range that contains the best models with 95% confidence, and the black line is the most likely best value.*\n\nSeveral results go against the habits of the time:\n\n1. **The best depth is very stable.** Whatever the compute, the best models have about 20 blocks (about 60 layers). The usual practice is to make the network deeper as compute grows.\n2. **No bottleneck needed.** The best models have $b=1$, with the middle layer as wide as the output.\n3. **The width multiplier between stages is about 2.5.** The common practice is to double the width at each stage.\n4. **Inverted bottlenecks and depthwise convolution both do worse.** The $b<1$ common in mobile networks makes results slightly worse, and depthwise convolution with $g=1$ is worse still. When the authors retested in the appendix at 6.4GF with 100 epochs, the inverted bottleneck fell even further behind.\n5. **A fixed resolution of $224\\times224$ is best.** This runs counter to EfficientNet's finding that scaling up the resolution helps, and it holds even at higher compute.\n\n$g$, $w_a$, and $w_0$, on the other hand, grow with compute. In addition, adding an SE module to the X block gives the RegNetY space, which is better still overall.\n\nThe authors also bring in a metric few people usually look at, **activations**, the total size of the output tensors of all convolutional layers. On a V100 GPU, inference time has a correlation coefficient of 0.897 with activations, but only 0.643 with FLOPs. Hardware like GPUs is often limited by memory bandwidth, so the size of the intermediate results predicts speed better than the number of multiply-adds does. Among the best models, activations grow roughly with the square root of compute, while parameters grow linearly.\n\nBased on these trends, the authors further constrain RegNetX with $b=1$, $d\\le40$, and $w_m\\ge2$, and limit parameters and activations, giving a faster, more memory-efficient space; all the comparisons that follow use this version.\n\n### Compared with ResNet and ResNeXt\n\nTo compare final models, the authors try 25 random parameter settings at each compute level, pick the best, and train it 5 times for 100 epochs. All models, including the ResNets and ResNeXts being compared, use the same basic training setup, with no regularization or training tricks beyond weight decay:\n\n| Model | GFLOPs | Parameters | Inference (ms) | Top-1 error |\n| --- | ---: | ---: | ---: | ---: |\n| ResNet-50 | 4.1 | 22.6M | 53 | 23.2% |\n| ResNeXt-50 | 4.2 | 25.0M | 78 | 21.9% |\n| **RegNetX-4.0GF** | 4.0 | 22.1M | 69 | **21.4%** |\n| ResNet-101 | 7.8 | 44.6M | 90 | 21.4% |\n| ResNeXt-101 | 8.0 | 44.2M | 137 | 20.7% |\n| **RegNetX-8.0GF** | 8.0 | 39.6M | 94 | **20.7%** |\n| ResNet-152 | 11.5 | 60.2M | 130 | 20.9% |\n| ResNeXt-152 | 11.7 | 60.0M | 197 | 20.4% |\n| **RegNetX-12GF** | 12.1 | 46.1M | 137 | **20.3%** |\n\n*Table: Excerpted from Table 3(b) of the paper, grouped by similar compute. Inference time is for one batch of 64 images on a V100.*\n\nIn all three groups, RegNetX is the best or tied for the best. In the 8GF group, RegNetX ties ResNeXt-101 but needs only 94 ms for inference, versus 137 ms for ResNeXt-101. These networks use the same components and training setup and differ only in structure, so the authors conclude that all of these gains come from the design space.\n\n### The Mobile Regime\n\nAround 600MF, RegNetX-600MF has a top-1 error of 25.9% and RegNetY-600MF 24.5%. At the same compute, MobileNet-V2 reports 25.3% and ShuffleNet V2 25.1%, while the NAS-found NASNet-A gets 26.0% and AmoebaNet-C 24.3%.\n\nThe competitors' numbers are those reported in their own papers, usually with longer training and assorted enhancement tricks; RegNet uses only the basic 100-epoch setup and still comes quite close. The authors hope these simple models can serve as baselines for future research.\n\n### Compared with EfficientNet\n\nLast comes EfficientNet, the strongest model of the time. The authors take EfficientNet's original architectures but retrain them with the same training setup as RegNet, tuning only the learning rate and weight decay separately for them:\n\n| Model | GFLOPs | Parameters | Activations | Inference (ms) | Training (hours) | Top-1 error |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| EfficientNet-B0 | 0.4 | 5.3M | 6.7M | 34 | 11.7 | **24.9%** |\n| RegNetY-400MF | 0.4 | 4.3M | 3.9M | 19 | 5.1 | 25.9% |\n| EfficientNet-B3 | 1.8 | 12.0M | 23.8M | 114 | 32.1 | 22.5% |\n| RegNetY-1.6GF | 1.6 | 11.2M | 8.0M | 39 | 10.1 | **22.0%** |\n| EfficientNet-B5 | 9.9 | 30.0M | 98.9M | 504 | 135.1 | 21.5% |\n| RegNetY-8.0GF | 8.0 | 39.2M | 18.0M | 113 | 28.1 | **20.1%** |\n\n*Table: Three groups excerpted from Table 4 of the paper. Both use the basic 100-epoch training setup with weight decay only; inference time is measured as in the previous table.*\n\nAt very low compute, EfficientNet is better; as compute goes up, RegNetY takes the lead. The speed gap is even more striking. EfficientNet-B5 takes 504 ms per inference batch, while RegNetY-8.0GF needs only 113 ms (by the time B5 finishes one batch, it has already run through four), with lower error to boot.\n\nThe speed gap ties back to activations. EfficientNet scales up both resolution and depth, so its activations grow roughly linearly with compute, reaching 98.9M for B5; RegNet's grow with the square root, only 18.0M at 8.0GF.\n\nKeep the training setup in mind when reading this table. The original EfficientNet paper used longer training and enhancements such as AutoAugment, and reported much lower errors, 16.7% for B5, for example. The authors' own experiments also show that the training setup alone can shift EfficientNet-B0's error by about 2 percentage points. So this table compares “which architecture is better under the same training conditions,” not the best results each can achieve.\n\n## Conclusion\n\nDo good architectures all look alike? Within this paper's design space, at least, the answer is yes. Narrowing all the way down from $10^{18}$ networks, the good models that remain all have widths on the same quantized line, a depth of about 20 blocks, no bottleneck, and widths that grow by a factor of about 2.5 from one stage to the next.\n\nThese regularities were found with a fixed X block, low compute, and short training, and the authors acknowledge that putting more options, such as operators, into the design space might lead to different answers.\n\nTo draw this line, the authors trained several thousand models just to compare design spaces. That's not cheap, but what they got in return is a set of rules you can actually explain, far easier to make sense of than a NAS-delivered network that amounts to “that's just how it is, don't ask.”\n","imageSizes":{"/content/papers/assets/526c9907c4f475ec129b14c60e3189345920cb53e55b91dea16ee61a5c64c0b2.png":{"width":1013,"height":387},"/content/papers/assets/a529e6e4b1fdd3d828f2fabfc86390e33c5501faa9cca45a206d01dc23706bf0.png":{"width":863,"height":467},"/content/papers/assets/477f84998189b88eda3ae1c093bfad949ef34cf53e00368019c348b971c3a63d.png":{"width":1013,"height":550},"/content/papers/assets/88e61c9524447ce29968da46650f265df2c5f591a45aaf965adef78e49826d51.png":{"width":1004,"height":409}}}