Back to paper library
arXiv 2021CVPR 2021ReparameterizationPaper notes

RepVGG

Making VGG Great Again

Original paper

RepVGG: Making VGG-style ConvNets Great Again

Paper authors

Xiaohan Ding · Xiangyu Zhang · Ningning Ma · Jungong Han · Guiguang Ding · Jian Sun

Publication
CVPR 2021Conference
First submitted to arXiv
arXiv:2101.03697
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{reparameterizationrepvgg2021,
  title = {{RepVGG: Making VGG-style ConvNets Great Again}},
  author = {Xiaohan Ding and Xiangyu Zhang and Ningning Ma and Jungong Han and Guiguang Ding and Jian Sun},
  year = {2021},
  eprint = {2101.03697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.03697}
}
On this page

Is it true that fewer parameters in a model result in faster speeds?

You might know the answer is: "Not necessarily!"

This is because we need to consider the computational load, specifically the FLOPs (floating-point operation count) of the model.

So let's ask another question: "Does fewer FLOPs in a model mean faster speed?"

The answer is still "Not necessarily!"

Defining the Problem

There are many factors influencing inference speed!

Many research papers focus on reducing the FLOPs of a model, but when used in engineering, the inference speed doesn't always improve, and sometimes it even slows down.

This is a typical gap between academia and industry, and as engineers, we can't be misled by the computational data.

High Memory Usage

With the advent of Inception, ResNet, and DenseNet, many researchers shifted their focus to carefully designed architectures, making models increasingly complex. Moreover, some architectures are based on NAS (Neural Architecture Search) or use compound scaling strategies.

Although these network architectures achieve higher accuracy, their complex branching leads to increased memory usage, as shown in the figure above.

In residual structures, branches require additional memory to store branch results during inference. If you have multiple branches (yes, Inception, looking at you!), the memory usage becomes even higher.

Slow Inference Speed

Some operations depend strongly on hardware and library support. MobileNet-style depthwise convolution reduces parameters and arithmetic, yet can run more slowly in some environments. The example below is configuration-specific, not a universal device comparison:

Similarly, the channel shuffle operation in ShuffleNet, although not computationally intensive, also reduces inference speed:

However...

Multi-branch structures are very effective!

Multi-branch architectures allow the model to become an implicit ensemble of many shallow models.

Residual connections provide a shortcut. For ResNet, theoretically, for NN blocks there are 2N2^N path choices. This implicit-ensemble interpretation counts 2N2^N paths through shared blocks.

Additionally, shortcuts offer better gradient flow, alleviating the training difficulty caused by vanishing gradients.

ResNet's success has already proven the effectiveness of residual connections, so why give them up?

Solving the Problem

Model Architecture

To address the above issues, the authors design a residual-inspired RepVGG block whose branches can be merged exactly. This is not a method for converting an arbitrary pretrained ResNet directly into VGG.

The authors redefine the RepVGG unit, as we need multi-branch architectures to improve model performance during training. Hence, the structure is as shown in Figure (b):

  1. A 3x3 convolution
  2. A 1x1 convolution
  3. An identity connection, present only when channels match and stride is 1

But during inference, we need to remove all branches, as shown in Figure (c).

The training structure is defined this way to satisfy the multi-branch training architecture while also meeting the model's inference requirements:

  • Reparameterization of the model.

Reparameterization

Removing branches isn't about directly deleting them; past research on model pruning has shown that direct deletion leads to performance degradation.

The authors propose a new method: Reparameterization.

In the RepVGG unit, each convolution operation is followed by Batch Normalization. With BN statistics fixed in evaluation mode, these affine branches can be merged into an equivalent convolution. ReLU follows the sum; separate ReLUs inside the branches would prevent this exact fusion.

The entire process is as follows:

  1. Merge the 3x3 convolution and BatchNorm into an equivalent convolution operation.
  2. Expand the 1x1 convolution into an equivalent 3x3 convolution using zero-padding, then merge it with BatchNorm into an equivalent convolution operation.
  3. Expand the identity connection into an equivalent 3x3 convolution, then merge it with BatchNorm into an equivalent convolution operation.
  4. Finally, add the three branches together into an equivalent convolution operation.

Although you might not like it, we still describe the specific process provided in the paper:

  • Let W(3)∈RC2×C1×3×3W^{(3)} \in \mathbb{R}^{C_2 \times C_1 \times 3 \times 3} represent the kernel of a 3×33 \times 3 convolution layer with C1C_1 input channels and C2C_2 output channels.
  • Let W(1)∈RC2×C1W^{(1)} \in \mathbb{R}^{C_2 \times C_1} represent the kernel of a 1×11 \times 1 convolution layer with C1C_1 input channels and C2C_2 output channels.
  • μ(3),σ(3),γ(3),β(3)\mu^{(3)}, \sigma^{(3)}, \gamma^{(3)}, \beta^{(3)} denote the mean, standard deviation, learnable scaling factor, and bias of the BN layer after the 3×33 \times 3 convolution layer.
  • μ(1),σ(1),γ(1),β(1)\mu^{(1)}, \sigma^{(1)}, \gamma^{(1)}, \beta^{(1)} denote the mean, standard deviation, learnable scaling factor, and bias of the BN layer after the 1×11 \times 1 convolution layer.
  • μ(0),σ(0),γ(0),β(0)\mu^{(0)}, \sigma^{(0)}, \gamma^{(0)}, \beta^{(0)} denote the identity branch.

Let M(1)∈RN×C1×H1×W1M^{(1)} \in \mathbb{R}^{N \times C_1 \times H_1 \times W_1} and M(2)∈RN×C2×H2×W2M^{(2)} \in \mathbb{R}^{N \times C_2 \times H_2 \times W_2} represent the input and pre-ReLU output, respectively, and ∗\ast denote convolution operation.

If C1=C2,H1=H2,W1=W2C_1 = C_2, H_1 = H_2, W_1 = W_2, then:

M(2)=bn⁡(M(1)∗W(3),μ(3),σ(3),γ(3),β(3))+bn⁡(M(1)∗W(1),μ(1),σ(1),γ(1),β(1))+bn⁡(M(1),μ(0),σ(0),γ(0),β(0)).\begin{aligned} M^{(2)} ={}& \operatorname{bn}(M^{(1)}\ast W^{(3)},\mu^{(3)},\sigma^{(3)},\gamma^{(3)},\beta^{(3)})\\ &+\operatorname{bn}(M^{(1)}\ast W^{(1)},\mu^{(1)},\sigma^{(1)},\gamma^{(1)},\beta^{(1)})\\ &+\operatorname{bn}(M^{(1)},\mu^{(0)},\sigma^{(0)},\gamma^{(0)},\beta^{(0)}). \end{aligned}

This notation assumes ungrouped convolution. The BN denominator σ\sigma includes numerical stabilization, running_var+ϵ\sqrt{\mathrm{running\_var}+\epsilon}. Grouped variants must preserve compatible groups; blocks that change channels or resolution omit the identity branch.

It should be noted that this conversion's equivalence requires that the 3x3 and 1x1 layers have the same stride and that the latter's padding configuration should be one pixel less than the former. For instance, if the 3x3 convolution layer has a padding of 1, the 1x1 convolution layer should have a padding of 0.

Parameter Configuration

Finally, the authors provide several different configurations of RepVGG. In the table above, Layers of each stage indicates the number of layers at different resolution features.

The first stage only has one layer because the image size is large, taking up a lot of computation. Most layers are concentrated in the fourth stage (layer count alone does not determine computation by stage), with 14 or 16 layers stacked.

The parameters a and b are used to control the number of channels, with the specific configuration shown in the table below:

Discussion

ImageNet Performance

In this study, the authors compared RepVGG with some classic and state-of-the-art models on the ImageNet-1K dataset.

These models include VGG-16, ResNet, ResNeXt, EfficientNet, and RegNet. The dataset comprises 1.28 million training images and 50,000 validation images.

During training, lightweight and medium models used simple data augmentation techniques such as random cropping and horizontal flipping, while heavyweight models added techniques such as warm-up, cosine learning rate annealing, label smoothing, mixup, and Autoaugment.

The authors trained on 8 GPUs, using the standard SGD optimizer and cosine annealing learning rate strategy.

Speed here is throughput in images/second on a single NVIDIA 1080Ti with batch size 128 and FP32, rather than single-image mobile latency. Results show that RepVGG outperforms many classic models in both accuracy and speed. For example, RepVGG-A0 improves accuracy by 1.25 percentage points and throughput by 33% over ResNet-18, and RepVGG-A1 improves accuracy by 0.29 percentage points and throughput by 64% over ResNet-34.

Even in comparisons with larger models such as ResNet-101 and ResNet-152, RepVGG models with interleaved grouped layers show significant advantages.

In terms of parameter efficiency, RepVGG also performs excellently. Compared to VGG-16, RepVGG-B2 reduces parameters by 42%, increases running speed by 10%, and improves accuracy by 6.57 percentage points. This demonstrates RepVGG's ability to maintain high performance with fewer parameters.

Finally, RepVGG also shows good performance compared to state-of-the-art baseline models. For example, RepVGG-A2 outperforms EfficientNetB0 by 1.37 percentage points in accuracy and is 59% faster. RepVGG-B1 outperforms RegNetX3.2GF by 0.39 percentage points in Top-1 accuracy and is slightly faster.

Ablation Experiments

To verify the importance of structural reparameterization technology, the authors conducted detailed experiments and ablation studies to explore its impact on model performance.

All models were trained from scratch for 120 epochs under the same simple training settings.

First, the authors conducted an ablation study by removing branches from each block of RepVGG-B0. The results showed that after removing two branches, the model's accuracy was only 72.39%. When keeping the 1x1 branch, accuracy increased to 73.15%; keeping the identity branch increased accuracy to 74.79%. The full-featured RepVGG-B0 achieved 75.14% accuracy.

The table measures inference using the unconverted training structures: full RepVGG-B0 processes 1061 images/s, while the version with only the 3×3 branch reaches 1810 images/s. After conversion these variants share the same inference architecture; the branches improve training outcomes. The 1061 images/s figure is not the full model's deployed throughput.


To further compare, the authors established a series of variants and baseline models, compared with RepVGG-B0, as shown in the table above.

These variants include:

  • Identity w/o BN: Removing BN from the identity branch.
  • Post-addition BN: Removing the BN layers from the three branches and adding a BN layer after addition.
  • Branch-wise+ReLU: Inserting ReLU into each branch. This prevents exact conversion to a single convolution, so it is only a comparison to study added nonlinearity.
  • DiracNet: Using a carefully designed convolution kernel reparameterization.
  • Trivial Re-param: Directly adding the identity kernel to the 3x3 kernel.
  • Asymmetric Convolution Block (ACB): Another form of structural reparameterization.
  • Residual Reorg: Reorganizing each stage similarly to ResNet.

The authors believe that the superiority of structural reparameterization over DiracNet and Trivial Re-param lies in the former relying on actual data flow through specific structures with batch-dependent nonlinear behavior during training (BN), while the latter only uses another mathematical expression of the convolution kernel. Specifically, removing BN reduces accuracy, while adding ReLU improves accuracy, indicating that non-linear components during training are crucial for performance.

Comparison with ACB shows that RepVGG's success is not just due to over-parameterization. RepVGG blocks can be equivalently converted into a single convolution for inference; training behavior depends on the branch structure. Comparison with Residual Reorg shows that RepVGG outperforms by 0.58 percentage points. The authors interpret the larger set of branch paths as a possible explanation.

These experimental results indicate that structural reparameterization technology significantly impacts model performance during training and inference and is a key method for training powerful ordinary ConvNets.

Conclusion

Structural reparameterization is one of the key technologies behind RepVGG's success.

This method allows the model to maintain high flexibility during training while merging each block’s branches into one convolution for inference and retaining the ReLU between blocks. The full network remains a multilayer nonlinear model. This design enables RepVGG to significantly improve inference speed while maintaining high accuracy.

Experimental results show that RepVGG outperforms several state-of-the-art models in terms of the trade-off between speed and accuracy, making it an ideal choice for high-performance computing and real-time applications.

RepVGG represents a new design approach for convolutional neural networks, demonstrating how to simplify design without sacrificing performance. Its simple and efficient structure and strong performance open new avenues for the development of convolutional neural networks.