{"notes":[{"id":"classic-cnns/deformable-convnets","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deformable ConvNets","subtitle":"Fancy Footwork","author":"Z. Yuan","paperDate":"2017-03-17","publishedOn":"2024-05-01","world":"external/deformable-convnets","paperTitle":"Deformable Convolutional Networks","paperUrl":"https://arxiv.org/abs/1703.06211","bibliography":{"title":"Deformable Convolutional Networks","authors":["Jifeng Dai","Haozhi Qi","Yuwen Xiong","Yi Li","Guodong Zhang","Han Hu","Yichen Wei"],"url":"https://arxiv.org/abs/1703.06211","pdfUrl":"https://arxiv.org/pdf/1703.06211v3","codeUrl":null,"preprint":{"id":"1703.06211","submittedOn":"2017-03-17","url":"https://arxiv.org/abs/1703.06211"},"publication":{"venue":"ICCV","year":2017,"kind":"conference","url":"https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html"},"citation":"@misc{classiccnnsdeformableconvnets2017,\n  title = {{Deformable Convolutional Networks}},\n  author = {Jifeng Dai and Haozhi Qi and Yuwen Xiong and Yi Li and Guodong Zhang and Han Hu and Yichen Wei},\n  year = {2017},\n  eprint = {1703.06211},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1703.06211}\n}\n","citationPath":"/content/papers/citations/classic-cnns/deformable-convnets.f724f3ff1784.bib"},"thumbnail":{"src":"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png","width":987,"height":633},"locale":"en","path":"/content/papers/en/classic-cnns/deformable-convnets.319cd37ba4c1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"the-fixed-grid","text":"The Fixed Grid","depth":3},{"id":"someone-already-taught-networks-to-warp","text":"Someone Already Taught Networks to Warp","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-two-steps-of-convolution","text":"The Two Steps of Convolution","depth":3},{"id":"an-offset-for-every-point","text":"An Offset for Every Point","depth":3},{"id":"sampling-at-fractional-positions","text":"Sampling at Fractional Positions","depth":3},{"id":"where-the-offsets-come-from","text":"Where the Offsets Come From","depth":3},{"id":"two-special-cases","text":"Two Special Cases","depth":3},{"id":"deformable-roi-pooling","text":"Deformable RoI Pooling","depth":3},{"id":"into-the-network","text":"Into the Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"receptive-fields-really-do-follow-the-object","text":"Receptive Fields Really Do Follow the Object","depth":3},{"id":"how-many-layers-to-replace","text":"How Many Layers to Replace","depth":3},{"id":"compared-with-atrous-convolution","text":"Compared with Atrous Convolution","depth":3},{"id":"the-cost","text":"The Cost","depth":3},{"id":"coco","text":"COCO","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"classic-cnns/stn","relation":"referenced","topic":""},{"id":"object-detection/deformable-detr","relation":"referenced","topic":""}]},{"id":"classic-cnns/stn","category":"classic-cnns","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"STN","subtitle":"Moving Heaven and Earth","author":"Z. Yuan","paperDate":"2015-06-05","publishedOn":"2023-10-25","world":"external/stn","paperTitle":"Spatial Transformer Networks","paperUrl":"https://arxiv.org/abs/1506.02025","bibliography":{"title":"Spatial Transformer Networks","authors":["Max Jaderberg","Karen Simonyan","Andrew Zisserman","Koray Kavukcuoglu"],"url":"https://arxiv.org/abs/1506.02025","pdfUrl":"https://arxiv.org/pdf/1506.02025v3","codeUrl":null,"preprint":{"id":"1506.02025","submittedOn":"2015-06-05","url":"https://arxiv.org/abs/1506.02025"},"publication":{"venue":"NeurIPS","year":2015,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2015/hash/33ceb07bf4eeb3da587e268d663aba1a-Abstract.html"},"citation":"@misc{classiccnnsstn2015,\n  title = {{Spatial Transformer Networks}},\n  author = {Max Jaderberg and Karen Simonyan and Andrew Zisserman and Koray Kavukcuoglu},\n  year = {2015},\n  eprint = {1506.02025},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1506.02025}\n}\n","citationPath":"/content/papers/citations/classic-cnns/stn.e823b15c17eb.bib"},"thumbnail":{"src":"/content/papers/assets/7b29191fd34e282fbd2d0fa901014f5770192cf1641d814b39b43a903817fb7f.png","width":480,"height":364},"locale":"en","path":"/content/papers/en/classic-cnns/stn.85f776fefe90.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"three-components","text":"Three Components","depth":3},{"id":"grid-generator-working-backward-from-the-output","text":"Grid Generator: Working Backward from the Output","depth":3},{"id":"sampler-taking-values-between-grid-points","text":"Sampler: Taking Values Between Grid Points","depth":3},{"id":"putting-it-into-a-network","text":"Putting It into a Network","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"distorted-mnist","text":"Distorted MNIST","depth":3},{"id":"street-view-house-numbers","text":"Street View House Numbers","depth":3},{"id":"fine-grained-bird-classification","text":"Fine-Grained Bird Classification","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"text-recognition/rare","relation":"referenced","topic":""}]},{"id":"object-detection/deformable-detr","category":"object-detection","legacyPath":"object-detection/2010-deformable-detr","legacySlug":"2010-deformable-detr","legacyPaths":[],"title":"Deformable DETR","subtitle":"Deformable Attention Mechanism","author":"Z. Yuan","paperDate":"2020-10-08","publishedOn":"2024-09-20","paperTitle":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paperUrl":"https://arxiv.org/abs/2010.04159","bibliography":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":["Xizhou Zhu","Weijie Su","Lewei Lu","Bin Li","Xiaogang Wang","Jifeng Dai"],"url":"https://arxiv.org/abs/2010.04159","pdfUrl":"https://arxiv.org/pdf/2010.04159","codeUrl":null,"preprint":{"id":"2010.04159","submittedOn":"2020-10-08","url":"https://arxiv.org/abs/2010.04159"},"publication":{"venue":"ICLR","year":2021,"kind":"conference","url":"https://arxiv.org/abs/2010.04159"},"citation":"@misc{objectdetectiondeformabledetr2020,\n  title = {{Deformable DETR: Deformable Transformers for End-to-End Object Detection}},\n  author = {Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},\n  year = {2020},\n  eprint = {2010.04159},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.04159}\n}\n","citationPath":"/content/papers/citations/object-detection/deformable-detr.df721a30472e.bib"},"thumbnail":{"src":"/content/papers/assets/36edfc5595a85c73fcdc9a4bbb9e1ca512e1f100676eb7c37203d71c84905c05.jpg","width":1224,"height":752},"locale":"en","path":"/content/papers/en/object-detection/deformable-detr.68972544ff9b.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"deformable-attention","text":"Deformable Attention","depth":3},{"id":"multi-scale-computation","text":"Multi-Scale Computation","depth":3},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"code-breakdown","text":"Code Breakdown","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-detr","text":"Comparison with DETR","depth":3},{"id":"ablation-studies","text":"Ablation Studies","depth":3},{"id":"sota-comparison","text":"SOTA Comparison","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"object-detection/detr","relation":"background","topic":"Set prediction and attention-based detection"},{"id":"object-detection/h-detr","relation":"next","topic":"Set prediction and attention-based detection"},{"id":"object-detection/smca-detr","relation":"comparison","topic":"Faster convergence for Transformer detectors"}]}],"text":"In the same photo, a nearby dog looks big and a faraway dog looks small; the same cat can be sitting, lying down, or stretching. Besides the categories themselves, a recognition model has to cope with geometric variations like these.\n\nConvolutional networks handle this mainly through data. During training, images are flipped, scaled, and cropped so the network sees enough variations, and a large enough model memorizes them. The network's own structure, though, is fixed. A $3\\times3$ convolution always samples on a square grid, pooling always downsizes by a fixed ratio, and RoI pooling always cuts a proposal into fixed bins.\n\nIn March 2017, Jifeng Dai and colleagues at Microsoft Research Asia released this paper. They wanted the sampling locations of convolution to move on their own.\n\n## Defining the Problem\n\n### The Fixed Grid\n\nThe authors note that there are traditionally two ways to handle geometric variation. One is data augmentation, making the training data cover enough variations; the other is designing features and algorithms that are insensitive to those variations, such as SIFT, or sliding-window detection.\n\nBoth rest on the same premise, that you know in advance which variations will occur. Data augmentation requires deciding beforehand how to rotate and how to scale; hand-crafted invariant features can't be built at all for variations that are too complex. As soon as a new task brings variations nobody anticipated, these approaches run out of options.\n\nConvolutional networks have the same problem, and it stems from their fixed structure. The authors give two examples:\n\n1. **Receptive fields in the same layer are all the same size.** High-level features carry semantics, and different positions may correspond to objects of different sizes and shapes, yet the receptive fields are identical. Tasks that need fine localization, such as semantic segmentation, would want receptive fields that adapt to the object.\n2. **Detection still extracts features from boxes.** The object detection methods of the time all ended up extracting features from rectangular proposals. For objects without a fixed shape, such as people and animals, most of the box is background.\n\n### Someone Already Taught Networks to Warp\n\nSTN had already let networks learn geometric transformations on their own. It predicts a set of global transformation parameters, such as an affine matrix, and then warps the whole feature map accordingly.\n\n\n> [!TIP]\n> \n> For STN, see our note:\n> \n> - **[[15.06] STN: Moving Heaven and Earth](/blog/papers/classic-cnns/stn)**\n\n\nSTN applies one transformation to the entire feature map; the computation is expensive, the transformation parameters are hard to learn, and its successes at the time were mainly in small-scale image classification. Detection and segmentation scenes contain many objects, each with its own size and pose, and a single global transformation can't take care of them all.\n\nCould each sampling point of a convolution, at every position, move on its own according to the content in front of it?\n\n## Solving the Problem\n\n### The Two Steps of Convolution\n\nLet's first spell out ordinary convolution. The authors break 2D convolution into two steps, first sampling the input feature map on a regular grid $\\mathcal{R}$, then multiplying the sampled values by the weights and summing them.\n\nFor a $3\\times3$ convolution, the grid is\n\n$$\n\\mathcal{R}=\\{(-1,-1),(-1,0),\\ldots,(0,1),(1,1)\\},\n$$\n\n9 points in total. For each position $\\mathbf{p}_0$ on the output feature map,\n\n$$\n\\mathbf{y}(\\mathbf{p}_0)=\\sum_{\\mathbf{p}_n\\in\\mathcal{R}}\\mathbf{w}(\\mathbf{p}_n)\\cdot\\mathbf{x}(\\mathbf{p}_0+\\mathbf{p}_n).\n$$\n\n$\\mathbf{p}_n$ runs through the 9 points of the grid in turn. This equation is the same for every channel, so from here on we look only at 2D spatial positions.\n\n### An Offset for Every Point\n\nDeformable convolution changes just one thing, adding an offset $\\Delta\\mathbf{p}_n$ to each point on the grid:\n\n$$\n\\mathbf{y}(\\mathbf{p}_0)=\\sum_{\\mathbf{p}_n\\in\\mathcal{R}}\\mathbf{w}(\\mathbf{p}_n)\\cdot\\mathbf{x}(\\mathbf{p}_0+\\mathbf{p}_n+\\Delta\\mathbf{p}_n).\n$$\n\nThe weights $\\mathbf{w}$ stay the same; what changes is where the values are taken from.\n\n[![Four diagrams of 3x3 convolution sampling locations on a dashed grid. (a) 9 green points in a regular 3x3 grid; (b) each light-green original location is pushed by a light-blue arrow to a new dark-blue location, in different directions and by different distances; (c) the 9 points are pushed outward from the center proportionally, forming an enlarged grid; (d) the 9 points are rotated together by an angle](/content/papers/assets/45479d40b2beb8096d042f56871e08317e36ffc2be1857a325732d436b36da0f.png)](/content/papers/assets/45479d40b2beb8096d042f56871e08317e36ffc2be1857a325732d436b36da0f.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1703.06211v3#page=2). (a) The regular sampling grid of standard convolution; (b) deformable convolution, with an offset added to each point; (c)(d) are special cases of (b), corresponding to scaling and rotation.*\n\n(a) is ordinary convolution. In (b), each of the 9 points is pushed to a new location, in different directions and by different distances. (c) and (d) are special cases of (b). In (c), all the points are pushed outward from the center, which amounts to enlarging the kernel; in (d), all the points rotate together around the center, which amounts to a rotation. In other words, transformations such as scaling, aspect ratio, and rotation are all contained in “each point gets its own offset.”\n\n### Sampling at Fractional Positions\n\nThe offsets are learned, so they usually aren't exactly whole numbers, and the sampling points land between grid points. The authors use bilinear interpolation to get the values:\n\n$$\n\\mathbf{x}(\\mathbf{p})=\\sum_{\\mathbf{q}}G(\\mathbf{q},\\mathbf{p})\\cdot\\mathbf{x}(\\mathbf{q}),\\qquad\nG(\\mathbf{q},\\mathbf{p})=g(q_x,p_x)\\cdot g(q_y,p_y),\\qquad g(a,b)=\\max(0,1-|a-b|).\n$$\n\n$\\mathbf{q}$ runs over all integer positions on the feature map, but $g$ is nonzero only when the distance is less than 1, so in practice only the four nearest points get any weight. Let's work through it on a $5\\times5$ feature map, with the center position $\\mathbf{p}_0=(2,2)$, the grid's top-right point $\\mathbf{p}_n=(-1,1)$, and a learned offset of, say, $(0.25,0.5)$:\n\n```python\nimport torch\n\nx = torch.arange(25.).reshape(5, 5)                # a 5×5 feature map whose values are simply 0 to 24\nprint(x)\n\ndef g(a, b):\n    return max(0.0, 1 - abs(a - b))\n\ndef sample(x, py, px):\n    \"\"\"Eqs. (3) and (4): weight every integer position q; only the four nearest points have nonzero weight\"\"\"\n    total = 0.0\n    for qy in range(x.shape[0]):\n        for qx in range(x.shape[1]):\n            w = g(qy, py) * g(qx, px)\n            if w > 0:\n                print(f\"  q=({qy},{qx}) value {x[qy, qx].item():4.1f} weight {w:.3f}\")\n                total += w * x[qy, qx].item()\n    return total\n\np0, pn, dp = (2, 2), (-1, 1), (0.25, 0.5)           # center position, top-right point of the grid, learned offset\npy, px = p0[0] + pn[0] + dp[0], p0[1] + pn[1] + dp[1]\nprint(f\"Sampling location p0 + pn + Δpn = ({py}, {px})\")\nprint(\"Interpolated value:\", sample(x, py, px))\n```\n\n```text\ntensor([[ 0.,  1.,  2.,  3.,  4.],\n        [ 5.,  6.,  7.,  8.,  9.],\n        [10., 11., 12., 13., 14.],\n        [15., 16., 17., 18., 19.],\n        [20., 21., 22., 23., 24.]])\nSampling location p0 + pn + Δpn = (1.25, 3.5)\n  q=(1,3) value  8.0 weight 0.375\n  q=(1,4) value  9.0 weight 0.375\n  q=(2,3) value 13.0 weight 0.125\n  q=(2,4) value 14.0 weight 0.125\nInterpolated value: 9.75\n```\n\nThe top-right point is originally at $(1,3)$, with value 8. After the offset, it lands at $(1.25,3.5)$; the four neighbors get weights according to their distance, and the four weights add up to 1. The values of this feature map are exactly $5\\times\\text{row}+\\text{column}$, and plugging in gives $5\\times1.25+3.5=9.75$, the same as the interpolated result.\n\nInterpolation is continuous, so the output is differentiable with respect to $\\Delta\\mathbf{p}_n$, and gradients can flow all the way back to the offsets.\n\n### Where the Offsets Come From\n\n[![Diagram of deformable convolution. On the left is the input feature map, marked with a set of 3x3 sampling boxes. A green arrow sends the input into conv, producing an offset field with 2N channels, from which the offsets for this position are taken, drawn as 3x3 small arrows pointing in different directions. The offsets are passed down to the deformable convolution, and the deformed sampling locations on the input feature map are connected by blue dashed lines to a single point on the output feature map on the right](/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png)](/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png)\n\n*Figure 2 from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/1703.06211v3#page=2). A $3\\times3$ deformable convolution.*\n\nThe offsets are computed by another convolutional layer from **the same input feature map**, which is the green path at the top of the figure. This convolution has the same kernel size and dilation as the main convolution, and its output offset field has the same spatial size as the input, with $2N$ channels. A $3\\times3$ convolution has $N=9$ sampling points, each needing two numbers, $(\\Delta y,\\Delta x)$, which makes 18 channels.\n\nIn other words, every position on the feature map has its own set of 9 offsets, and those offsets are computed by looking at the content around that position. This is what the authors mean by local, dense, and adaptive.\n\nLet's build a complete layer with torchvision's `deform_conv2d`:\n\n```python\nimport torch\nimport torch.nn as nn\nfrom torchvision.ops import deform_conv2d\n\nclass DeformableConv(nn.Module):\n    def __init__(self, cin, cout, k=3):\n        super().__init__()\n        self.weight = nn.Parameter(torch.randn(cout, cin, k, k) * 0.01)\n        # convolution that predicts the offsets: 3×3 like the main convolution, 2N = 18 output channels, initialized to 0\n        self.offset = nn.Conv2d(cin, 2 * k * k, k, padding=k // 2)\n        nn.init.zeros_(self.offset.weight)\n        nn.init.zeros_(self.offset.bias)\n\n    def forward(self, x):\n        offset = self.offset(x)                        # [B, 18, H, W]: 9 (Δy, Δx) pairs for each output position\n        print(\"Offset shape:\", tuple(offset.shape))\n        return deform_conv2d(x, offset, self.weight, padding=1)\n\nm = DeformableConv(256, 256)\ny = m(torch.randn(2, 256, 38, 38))\nprint(\"Output shape:\", tuple(y.shape))\nprint(f\"Main convolution parameters {m.weight.numel():,}, offset convolution parameters {sum(p.numel() for p in m.offset.parameters()):,}\")\n```\n\n```text\nOffset shape: (2, 18, 38, 38)\nOutput shape: (2, 256, 38, 38)\nMain convolution parameters 589,824, offset convolution parameters 41,490\n```\n\nThe offset convolution adds only about 7% of the main convolution's parameters. Its weights are initialized to 0, so at the start of training all the offsets are 0 and the layer is just an ordinary convolution; the offset convolution's learning rate is $\\beta$ times that of the other layers, with $\\beta=1$ by default. The main convolution's weights and the offsets are learned together during training, with no extra annotation required.\n\n### Two Special Cases\n\nThe proportional enlargement in Figure 1(c) is in fact atrous convolution. Let's verify the two special cases directly:\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom torchvision.ops import deform_conv2d\n\ntorch.manual_seed(0)\nx = torch.randn(1, 8, 7, 7)\nw = torch.randn(16, 8, 3, 3)\n\n# 1. All offsets 0: same as an ordinary 3×3 convolution\noffset = torch.zeros(1, 2 * 9, 7, 7)                # 9 sampling points, a (Δy, Δx) for each\ndiff = (deform_conv2d(x, offset, w, padding=1) - F.conv2d(x, w, padding=1)).abs().max()\nprint(\"Offsets all 0, difference from ordinary convolution:\", diff.item())\n\n# 2. Push each sampling point out by one cell: same as atrous convolution with dilation=2\ngrid = [(i, j) for i in (-1, 0, 1) for j in (-1, 0, 1)]\noffset = torch.tensor([v for i, j in grid for v in (i, j)], dtype=torch.float32)\noffset = offset.view(1, 18, 1, 1).expand(1, 18, 7, 7)\ndiff = (deform_conv2d(x, offset, w, padding=1) - F.conv2d(x, w, padding=2, dilation=2)).abs().max()\nprint(\"Offsets equal to grid coordinates, difference from dilation=2:\", diff.item())\n```\n\n```text\nOffsets all 0, difference from ordinary convolution: 0.0\nOffsets equal to grid coordinates, difference from dilation=2: 0.0\n```\n\nIn the second example, a grid point $(i,j)$ plus the offset $(i,j)$ moves to $(2i,2j)$, which is exactly atrous convolution with a dilation of 2. Atrous convolution's enlargement factor has to be hard-coded in advance, whereas deformable convolution lets the network decide for itself at every position.\n\n### Deformable RoI Pooling\n\nThe same idea is applied to RoI pooling in detection. RoI pooling cuts a proposal into $k\\times k$ bins, averages each bin, and outputs features of a fixed size. Deformable RoI pooling adds an offset to each bin so the bins can move onto the object.\n\nThe offsets are computed a little differently here. The authors first run an ordinary RoI pooling, followed by a fully connected layer that outputs “normalized” offsets $\\Delta\\hat{\\mathbf{p}}_{ij}$, which are then multiplied by the proposal's width and height to get the actual offsets:\n\n$$\n\\Delta\\mathbf{p}_{ij}=\\gamma\\cdot\\Delta\\hat{\\mathbf{p}}_{ij}\\circ(w,h),\\qquad \\gamma=0.1.\n$$\n\nMultiplying by the width and height makes offset learning independent of the proposal's size. The position-sensitive RoI pooling used in R-FCN also has a deformable version, in which the offsets are instead computed over the whole image by a convolutional layer.\n\n### Into the Network\n\nThe authors put deformable convolution into two backbones pretrained on ImageNet, namely ResNet-101 and Aligned-Inception-ResNet, which fixes a feature-alignment problem. As was common practice at the time, the stride of the last stage (conv5) is changed from 2 to 1 and its convolutions to dilation 2, raising the resolution of the output feature map from 1/32 to 1/16.\n\nDeformable convolution replaces only the last few $3\\times3$ convolutions in conv5, the last 3 by default. The backbones feed four task networks, namely DeepLab for semantic segmentation, the single-stage class-aware RPN, and the two-stage detectors Faster R-CNN and R-FCN. The latter two can also swap their RoI pooling for the deformable version.\n\n## Discussion\n\nSemantic segmentation uses PASCAL VOC and Cityscapes, measured by mIoU; detection uses PASCAL VOC and COCO, measured by mAP.\n\n### Receptive Fields Really Do Follow the Object\n\n[![Diagram of the receptive fields of two convolutional layers, over a grayscale photo of a small sheep and a large sheep, stacked in three levels. (a) standard convolution: on the top level there is one point on the small sheep and one on the large sheep, and the sampling points they map to in the two levels below are fixed-size square grids. (b) deformable convolution: for the same two points, the sampling points on the small sheep's side cluster on the small sheep, while those on the large sheep's side spread out widely, covering the whole large sheep](/content/papers/assets/e05ab32de27fdd8551794ded9ab71fafae0ea423979ef7250f9e7d31cedf1534.png)](/content/papers/assets/e05ab32de27fdd8551794ded9ab71fafae0ea423979ef7250f9e7d31cedf1534.png)\n\n*Figure 5 from the paper, excerpted from [**page 4 of the v3 PDF**](https://arxiv.org/pdf/1703.06211v3#page=4). Receptive fields of two layers of standard convolution (a) and two layers of deformable convolution (b).*\n\nTrace a point on the top level downward through two $3\\times3$ convolutions, and it maps to a cluster of sampling points on the bottom level. (a) is ordinary convolution, where the sampling region is the same-sized square whether the point sits on the small sheep or the large one. In (b), with deformable convolution, the point on the small sheep looks only at the small sheep, while the point on the large sheep spreads out over the whole animal.\n\nThis is an illustration, and the authors also quantified it. They define a metric called effective dilation, the mean distance between adjacent sampling points within a filter, which can be read as a rough measure of receptive field size. They then run R-FCN with 3 deformable convolution layers on the VOC 2007 test set and group the statistics by the kind of object on which the filter's center falls:\n\n| Layer | Small objects | Medium objects | Large objects | Background |\n| --- | ---: | ---: | ---: | ---: |\n| res5c | 5.3 ± 3.3 | 5.8 ± 3.5 | 8.4 ± 4.5 | 6.2 ± 3.0 |\n| res5b | 2.5 ± 1.3 | 3.1 ± 1.5 | 5.1 ± 2.5 | 3.2 ± 1.2 |\n| res5a | 2.2 ± 1.2 | 2.9 ± 1.3 | 4.2 ± 1.6 | 3.1 ± 1.1 |\n\n*Table: Excerpted from Table 2 of the paper, mean and standard deviation of effective dilation. Objects are divided into small, medium, and large by box area, with boundaries at $96^2$ and $224^2$ pixels.*\n\nAll three layers show the same trend, with the sampling points spreading wider as objects get larger. The background values fall between those of medium and large objects, which the authors take to mean that recognizing background also needs a fairly large receptive field.\n\n### How Many Layers to Replace\n\n| Deformable convolution layers | DeepLab (Cityscapes) | class-aware RPN | Faster R-CNN | R-FCN |\n| --- | ---: | ---: | ---: | ---: |\n| 0 (baseline) | 70.4 | 44.9 | 62.1 | 61.8 |\n| 1 | 73.5 | 54.4 | 63.8 | 63.0 |\n| 2 | 74.4 | 56.3 | 63.3 | 63.8 |\n| 3 (default) | **75.2** | 57.2 | 63.3 | 64.7 |\n| 6 | 75.1 | **57.7** | **64.0** | **65.4** |\n\n*Table: Excerpted from Table 1 of the paper, ResNet-101. DeepLab is mIoU on Cityscapes; the three detectors are mAP@0.7 on the VOC 2007 test set.*\n\nReplacing just the last layer improves DeepLab by 3 percentage points and class-aware RPN by nearly 10. These two methods have no RoI pooling and compute their results directly from convolutional features, which may be why they benefit the most. DeepLab saturates at 3 layers, while the detectors are still improving at 6, and the authors settle on 3 layers as a trade-off across tasks.\n\n### Compared with Atrous Convolution\n\nIf atrous convolution is a special case, would simply turning up the dilation have the same effect?\n\n| Setting | DeepLab (VOC / Cityscapes) | class-aware RPN | Faster R-CNN | R-FCN |\n| --- | ---: | ---: | ---: | ---: |\n| Atrous convolution (2,2,2), default | 69.7 / 70.4 | 68.0 / 44.9 | 78.1 / 62.1 | 80.0 / 61.8 |\n| Atrous convolution (4,4,4) | 73.1 / 71.9 | 72.8 / 53.1 | 78.6 / 63.1 | 80.5 / 63.0 |\n| Atrous convolution (6,6,6) | 73.6 / 72.7 | 73.6 / 55.2 | 78.5 / 62.3 | 80.2 / 63.5 |\n| Atrous convolution (8,8,8) | 73.2 / 72.4 | 73.2 / 55.1 | 77.8 / 61.8 | 80.3 / 63.2 |\n| Deformable convolution | **75.3 / 75.2** | **74.5 / 57.2** | 78.6 / 63.3 | 81.4 / 64.7 |\n| Deformable RoI pooling | – | – | 78.3 / 66.6 | 81.2 / 65.0 |\n| Both | – | – | **79.3 / 66.9** | **82.6 / 68.5** |\n\n*Table: Excerpted from Table 3 of the paper, ResNet-101. DeepLab is mIoU; the detectors are mAP@0.5 / mAP@0.7 on the VOC 2007 test set. The numbers in parentheses are the dilations of the last three $3\\times3$ convolutions.*\n\nIncreasing the dilation improves every task, which shows the default receptive field was too small to begin with. But the best dilation varies by task, 6 for DeepLab and 4 for Faster R-CNN, and no single value suits them all. Deformable convolution is no worse than the best dilation on any task, and it needs no per-task tuning.\n\nDeformable RoI pooling on its own already lifts both detectors' mAP@0.7 considerably, and this stricter metric demands more precise boxes. With both modules, R-FCN goes from 80.0 / 61.8 to 82.6 / 68.5.\n\n### The Cost\n\n| Method | Parameters | Network forward (s) | Total runtime (s) |\n| --- | ---: | ---: | ---: |\n| DeepLab (Cityscapes) | 46.0M | 0.610 | 0.650 |\n| + deformable modules | 46.1M | 0.656 | 0.696 |\n| Faster R-CNN | 58.3M | 0.147 | 0.190 |\n| + deformable modules | 59.9M | 0.192 | 0.234 |\n| R-FCN | 47.1M | 0.143 | 0.170 |\n| + deformable modules | 49.5M | 0.169 | 0.193 |\n\n*Table: Three groups excerpted from Table 4 of the paper, ResNet-101, on an Intel E5-2650 v2 CPU and an NVIDIA K40 GPU. Total runtime includes image resizing and post-processing such as NMS.*\n\nParameters go up by at most 2.4M, from which the authors conclude that the gains come from the ability to model geometric transformations, not from a bigger model. In time, DeepLab adds only about 7%, while Faster R-CNN's forward time goes from 0.147 seconds to 0.192 seconds, about 30% more.\n\nThe other cost is in deployment. Deformable convolution isn't a standard convolution; early on, exporting torchvision's `deform_conv2d` to ONNX required a [third-party package](https://pypi.org/project/deform-conv2d-onnx-exporter/) to break it down into ordinary operations, which also affected speed.\n\n### COCO\n\nFinally, a full comparison on COCO test-dev, measured by mAP@[0.5:0.95]:\n\n| Method | Backbone | Standard | Deformable |\n| --- | --- | ---: | ---: |\n| class-aware RPN | ResNet-101 | 23.2 | 25.8 |\n| Faster R-CNN | ResNet-101 | 29.4 | 33.1 |\n| R-FCN | ResNet-101 | 30.8 | 34.5 |\n| Faster R-CNN | Aligned-Inception-ResNet | 30.8 | 34.1 |\n| R-FCN | Aligned-Inception-ResNet | 32.9 | 36.1 |\n| R-FCN, multi-scale testing and box averaging | Aligned-Inception-ResNet | 35.5 | **37.5** |\n\n*Table: Excerpted from Table 5 of the paper.*\n\nOn ResNet-101, the three methods improve by 11%, 13%, and 12% in relative terms. The gains hold up with a stronger backbone or with tricks such as multi-scale testing. Broken down by object size, the largest absolute gains are on medium and large objects; Faster R-CNN's mAP on large objects, for example, goes from 47.1 to 51.2, and on small objects from 9.0 to 11.6.\n\n## Conclusion\n\nDeformable convolution adds a learned offset to every sampling point of a convolution. The change is small, adding only a few percent more parameters, yet the receptive field can adapt to the size and shape of objects, giving consistent gains in both segmentation and detection, especially on mAP@0.7, which demands more precise boxes.\n\nTo try it in your own network, torchvision's `deform_conv2d` plus a zero-initialized offset convolution is all it takes. Deployment isn't as easy, since it isn't a standard convolution and inference frameworks may not support it. The sampling points have mastered some fancy footwork, and the engineers deploying them may have to learn a few steps too.\n\nThis idea went a long way afterward. v2 added modulation weights to the sampling points, InternImage built a large-model backbone on v3, and [Deformable DETR](/blog/papers/object-detection/deformable-detr) carried it over to attention. We'll run into it again in the next few notes.\n","imageSizes":{"/content/papers/assets/45479d40b2beb8096d042f56871e08317e36ffc2be1857a325732d436b36da0f.png":{"width":988,"height":316},"/content/papers/assets/942dcd98b1119ce95e8a9a24b84ed42b780d062caafa004c66f75640c4e2584a.png":{"width":987,"height":633},"/content/papers/assets/e05ab32de27fdd8551794ded9ab71fafae0ea423979ef7250f9e7d31cedf1534.png":{"width":1000,"height":571}}}