{"notes":[{"id":"contrastive-learning/dino","category":"contrastive-learning","legacyPath":"contrastive-learning/2104-dino","legacySlug":"2104-dino","legacyPaths":[],"title":"DINO","subtitle":"Self-Distillation With No Labels","author":"Z. Yuan","paperDate":"2021-04-29","publishedOn":"2025-03-18","paperTitle":"Emerging Properties in Self-Supervised Vision Transformers","paperUrl":"https://arxiv.org/abs/2104.14294","bibliography":{"title":"Emerging Properties in Self-Supervised Vision Transformers","authors":["Mathilde Caron","Hugo Touvron","Ishan Misra","Hervé Jégou","Julien Mairal","Piotr Bojanowski","Armand Joulin"],"url":"https://arxiv.org/abs/2104.14294","pdfUrl":"https://arxiv.org/pdf/2104.14294","codeUrl":null,"preprint":{"id":"2104.14294","submittedOn":"2021-04-29","url":"https://arxiv.org/abs/2104.14294"},"publication":{"venue":"ICCV","year":2021,"kind":"conference","url":"https://doi.org/10.1109/iccv48922.2021.00951"},"citation":"@misc{contrastivelearningdino2021,\n  title = {{Emerging Properties in Self-Supervised Vision Transformers}},\n  author = {Mathilde Caron and Hugo Touvron and Ishan Misra and Hervé Jégou and Julien Mairal and Piotr Bojanowski and Armand Joulin},\n  year = {2021},\n  eprint = {2104.14294},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2104.14294}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/dino.20875c1e301c.bib"},"thumbnail":{"src":"/content/papers/assets/c2d0ab75afb02750e17183e186f29b0ef6e184eda12673a31c96f199e338188f.jpg","width":664,"height":642},"locale":"en","path":"/content/papers/en/contrastive-learning/dino.df5756884ee0.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"preventing-model-collapse","text":"Preventing Model Collapse","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"importance-of-different-components","text":"Importance of Different Components","depth":3},{"id":"model-collapse-testing","text":"Model Collapse Testing","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dinov2","relation":"next","topic":"Self-distillation for visual representations"}]},{"id":"contrastive-learning/dinov2","category":"contrastive-learning","legacyPath":"contrastive-learning/2304-dinov2","legacySlug":"2304-dinov2","legacyPaths":[],"title":"DINOv2","subtitle":"Universal Feature Forging Method","author":"Z. Yuan","paperDate":"2023-04-14","publishedOn":"2025-07-01","paperTitle":"DINOv2: Learning Robust Visual Features without Supervision","paperUrl":"https://arxiv.org/abs/2304.07193","bibliography":{"title":"DINOv2: Learning Robust Visual Features without Supervision","authors":["Maxime Oquab","Timothée Darcet","Théo Moutakanni","Huy V. Vo","Marc Szafraniec","Vasil Khalidov","Pierre Fernandez","Daniel Haziza","Francisco Massa","Alaaeldin El-Nouby","Mahmoud Assran","Nicolas Ballas","Wojciech Galuba","Russell Howes","Po-Yao Huang","Shang-Wen Li","Ishan Misra","Michael Rabbat","Vasu Sharma","Gabriel Synnaeve","Hu Xu","Hervé Jégou","Julien Mairal","Patrick Labatut","Armand Joulin","Piotr Bojanowski"],"url":"https://arxiv.org/abs/2304.07193","pdfUrl":"https://arxiv.org/pdf/2304.07193","codeUrl":"https://github.com/facebookresearch/dinov2","preprint":{"id":"2304.07193","submittedOn":"2023-04-14","url":"https://arxiv.org/abs/2304.07193"},"publication":{"venue":"Transactions on Machine Learning Research","shortVenue":"TMLR","year":2024,"kind":"journal","url":"https://openreview.net/forum?id=a68SUt6zFt"},"citation":"@misc{contrastivelearningdinov22023,\n  title = {{DINOv2: Learning Robust Visual Features without Supervision}},\n  author = {Maxime Oquab and Timothée Darcet and Théo Moutakanni and Huy V. Vo and Marc Szafraniec and Vasil Khalidov and Pierre Fernandez and Daniel Haziza and Francisco Massa and Alaaeldin El-Nouby and Mahmoud Assran and Nicolas Ballas and Wojciech Galuba and Russell Howes and Po-Yao Huang and Shang-Wen Li and Ishan Misra and Michael Rabbat and Vasu Sharma and Gabriel Synnaeve and Hu Xu and Hervé Jégou and Julien Mairal and Patrick Labatut and Armand Joulin and Piotr Bojanowski},\n  year = {2023},\n  eprint = {2304.07193},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2304.07193}\n}\n","citationPath":"/content/papers/citations/contrastive-learning/dinov2.e904e3854892.bib"},"thumbnail":{"src":"/content/papers/assets/9c68ef105804d4462aee8950a1bfd9f0929511f494b1a29690d857bf931da892.jpg","width":1224,"height":368},"locale":"en","path":"/content/papers/en/contrastive-learning/dinov2.2d46c57f4666.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"data-collection","text":"Data Collection","depth":2},{"id":"1-preparing-the-data-pool","text":"1. Preparing the Data Pool","depth":3},{"id":"2-removing-duplicates","text":"2. Removing Duplicates","depth":3},{"id":"3-self-supervised-retrieval","text":"3. Self-Supervised Retrieval","depth":3},{"id":"engineering-scale-and-efficiency","text":"Engineering Scale and Efficiency","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"image-level-feature-alignment","text":"Image-Level Feature Alignment","depth":3},{"id":"patch-level-prediction-task","text":"Patch-Level Prediction Task","depth":3},{"id":"head-design","text":"Head Design","depth":3},{"id":"sinkhorn-knopp-centering","text":"Sinkhorn-Knopp Centering","depth":3},{"id":"koleo-regularization","text":"KoLeo Regularization","depth":3},{"id":"high-resolution-training-phase","text":"High-Resolution Training Phase","depth":3},{"id":"engineering-implementation","text":"Engineering Implementation","depth":2},{"id":"flashattention","text":"FlashAttention","depth":3},{"id":"sequence-packing","text":"Sequence Packing","depth":3},{"id":"stochastic-depth","text":"Stochastic Depth","depth":3},{"id":"distributed-training","text":"Distributed Training","depth":3},{"id":"model-distillation","text":"Model Distillation","depth":3},{"id":"ablation-experiments","text":"Ablation Experiments","depth":2},{"id":"model-training-process","text":"Model Training Process","depth":3},{"id":"pretraining-data","text":"Pretraining Data","depth":3},{"id":"importance-of-loss-components","text":"Importance of Loss Components","depth":3},{"id":"knowledge-distillation-effect","text":"Knowledge Distillation Effect","depth":3},{"id":"high-resolution-training","text":"High-Resolution Training","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"imagenet-classification","text":"ImageNet Classification","depth":3},{"id":"video-classification","text":"Video Classification","depth":3},{"id":"instance-recognition","text":"Instance Recognition","depth":3},{"id":"semantic-segmentation","text":"Semantic Segmentation","depth":3},{"id":"qualitative-results","text":"Qualitative Results","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"contrastive-learning/dino","relation":"background","topic":"Self-distillation for visual representations"}]}],"text":"DINO appeared around the same time as MoCo v3, but its reported results look much stronger.\n\nThe name DINO comes from self-**di**stillation with **no** labels. I read it like the dinosaur name, DI-NO rather than Dee-No.\n\n\n> [!TIP]\n> \n> This judgment is based on the presence of various little dinosaurs on their Github...\n> \n> - [**https://github.com/facebookresearch/dino**](https://github.com/facebookresearch/dino)\n\n\n## Defining the Problem\n\nDo you remember BYOL?\n\nIf not, no worries, here’s a brief introduction. The BYOL architecture looks like this:\n\n\n\n![byol arch](/content/papers/assets/b8cc1c529a1c65a2a6ce495941153b6b24def2bccc29907a0a5edf64b16570c9.jpg)\n\n\n\nBYOL is a prominent demonstration of the idea that negative samples are \"unnecessary\" in the process of representation learning.\n\nThe core components of BYOL are three:\n\n1. **Siamese networks**: An online network and a target network, where the target network updates its parameters through exponential moving average (EMA).\n2. **Stop gradient**: The target network does not propagate gradients during training.\n3. **Additional MLP layer**: The only difference between the online and target networks is that the online network has an additional MLP for prediction.\n\n\n> [!TIP]\n> \n> If you are interested in the details of BYOL, you can refer to the paper we previously read:\n> \n> - [**[20.06] BYOL: No Need for Negative Samples**](/blog/papers/contrastive-learning/byol)\n\n\nIn this paper, the authors were inspired by BYOL and decided to make slight modifications to this structure.\n\nThe changes are minimal, just two:\n\n1. **Change the loss function**: BYOL uses MSE, while here it is changed to cross-entropy between teacher and student output distributions.\n2. **Remove the additional MLP layer**: This means the siamese network architecture here is exactly the same for both networks.\n\nIt may seem like not much, but this form is not without reasoning. The authors based it on this paper:\n\n- [**[17.03] Mean Teachers Are Better Role Models**](https://arxiv.org/abs/1703.01780)\n\nMean Teacher provides empirical support for weight-averaged teachers in its semi-supervised setting. It is relevant motivation, not a theorem that EMA teachers outperform every fixed teacher.\n\n\n> [!NOTE]\n> \n> Knowledge distillation is a training method in which we aim to have the student network mimic the output of the teacher network.\n\n\n## Solving the Problem\n\n\n\n![model arch](/content/papers/assets/c2d0ab75afb02750e17183e186f29b0ef6e184eda12673a31c96f199e338188f.jpg)\n\n\n\nAfter modifying the BYOL architecture, it looks like the figure above.\n\nGiven an input image $x$, the two networks each output a $K$-dimensional feature vector, which is converted to a probability distribution via the softmax function. For the student network (left side of the image), the probability for the $i$-th dimension, $P_s(x)(i)$, is defined as:\n\n$$\nP_s(x)^{(i)} = \\frac{\\exp\\left(\\frac{g_{\\theta_s}(x)^{(i)}}{\\tau_s}\\right)}{\\sum_{k=1}^{K} \\exp\\left(\\frac{g_{\\theta_s}(x)^{(k)}}{\\tau_s}\\right)}\n$$\n\nWhere $\\tau_s > 0$ is the temperature parameter used to control the sharpness of the output distribution. Similarly, for the teacher network (right side of the image), the probability distribution $P_t$ is defined using a temperature $\\tau_t$.\n\nThe teacher network $g_{\\theta_t}$ receives no gradient from this loss. We want the student distribution $P_s(x)$ to match the teacher distribution $P_t(x)$ by minimizing their cross-entropy:\n\n$$\n\\min_{\\theta_s} H(P_t(x), P_s(x))\n$$\n\nWhere the cross-entropy $H(a, b) = -\\sum a \\log b$ measures the difference between two distributions.\n\nSince self-supervised learning does not use labels, the authors employ a common technique in contrastive learning, where different augmentations of the same image are used to generate different \"views.\"\n\nIn this architecture, **all views are input to the student network, and only global views are input to the teacher network**. This design encourages the \"local-to-global\" correspondence.\n\nFor each global view $x \\in \\{x^g_1, x^g_2\\}$ and other views processed by the student network $x'$ (where $x' \\neq x$), the cross-entropy loss is calculated, and the total loss is minimized:\n\n$$\n\\min_{\\theta_s} \\sum_{x \\in \\{x^g_1, x^g_2\\}} \\sum_{\\substack{x' \\in V \\\\ x' \\neq x}} H(P_t(x), P_s(x'))\n$$\n\nThis loss function is general for the number of views, and can be used even with just two views; the standard setting in this paper uses two global views and multiple local views as basic parameters.\n\nThe student network's parameters $\\theta_s$ are optimized by minimizing the above loss function using gradient-based optimization; the reported ViT setup uses AdamW.\n\nThe teacher network's parameters $\\theta_t$ are updated using EMA from the student network's parameters $\\theta_s$, with the update rule as follows:\n\n$$\n\\theta_t \\leftarrow \\lambda \\theta_t + (1 - \\lambda) \\theta_s\n$$\n\nThis update method is similar to the momentum encoder, where the coefficient $\\lambda$ follows a cosine schedule from 0.996 to 1 during training.\n\n### Preventing Model Collapse\n\nThe issue with this type of siamese network self-supervised method is that the model often collapses unexpectedly. Previous studies have mostly used contrastive loss, clustering, and other methods to prevent the model from collapsing into a trivial solution. However, the authors here adopted centering and sharpening strategies.\n\n- **Centering:**\n\n  The centering operation subtracts the running center $c$ from the teacher logits of the teacher network, defined as:\n\n  $$\n  g_t(x) \\leftarrow g_t(x) - c\n  $$\n\n  This prevents any particular dimension from overly dominating the output, while simultaneously encouraging a more uniform distribution. Additionally, to ensure that the centering operation works stably under different batch sizes, the center $c$ is updated using exponential moving average:\n\n  $$\n  c \\leftarrow m c + (1 - m) \\frac{1}{B} \\sum_{i=1}^{B} g_{\\theta_t}(x_i)\n  $$\n\n  where $0 \\le m < 1$ is the EMA momentum coefficient and $B$ is the batch size. This update ensures that the center $c$ only depends on the first-order statistics of the batch.\n\n  The paper's prose writes the bias as $+c$, but its Algorithm 1 and the authors' implementation subtract the center updated from the batch mean.\n\n- **Sharpening:**\n\n  The sharpening operation is achieved by using a lower temperature $\\tau_t$ in the teacher's softmax normalization, which makes the teacher’s output probability distribution sharper. This balances the uniform tendency caused by centering, helping to prevent model collapse.\n\n## Discussion\n\n### Comparison with Other Methods\n\n\n\n![model compare](/content/papers/assets/470e3d3a53b5913cc5c859a840bda8edd82b7430f1d787a5ae2f360e267d6440.jpg)\n\n\n\nBased on experimental results, the authors categorized the comparisons into \"same architecture\" and \"cross-architecture\" comparisons:\n\n1. **Comparison within the Same Architecture:**\n\n   - **ResNet-50 Architecture:**\n     DINO achieves results on ResNet-50 that are comparable to the best existing self-supervised methods, verifying that DINO can perform on par with other methods under standard settings.\n   - **ViT-S Architecture:**\n     When using ViT-S (designed based on DeiT-S), DINO’s performance significantly surpasses BYOL, MoCov2, and SwAV.\n     - Linear classification reaches 77.0%, 3.5 percentage points above SwAV's 73.5% for ViT-S in the table.\n     - k-NN reaches 74.5%, 7.9 points above BYOL's 66.6% for ViT-S in the table.\n       DINO ViT-S's k-NN result (74.5%) is just 2.5 points below its linear result (77.0%). Such a small gap is not seen for the other listed methods or DINO ResNet-50 in this table.\n\n2. **Cross-Architecture Comparison:**\n\n   This section primarily explores the extreme performance of DINO-trained ViT models across different scale architectures, rather than directly comparing various methods.\n\n   Larger ViTs improve performance, while reducing patch size (the “/8” variants) has an even clearer effect in this table. For a given ViT architecture, smaller patches do not add parameters, but they create longer token sequences, increase memory use, and reduce inference throughput.\n\n   DINO's ViT-B/8 reaches 80.1% linear Top-1 and 77.4% k-NN accuracy. Its extractor has 85M parameters versus 794M for SimCLR v2 RN152w3+SK, with throughput of 63 versus 46 images/s. This is a cross-architecture comparison.\n\nDINO and SwAV both reach 75.3% linear accuracy with standard ResNet-50. With ViTs, smaller patches improve the reported accuracy while lowering throughput for the same architecture.\n\n\n> [!TIP]\n> \n> When we first read this part of the paper, we were quite confused.\n> \n> From the table, the higher the value of im/s, the faster the model's inference speed. For example, the standard ViT-S model (with a larger patch size) has 1007 im/s, while ViT-S/8 (with a patch size of 8×8) drops to 180 im/s. This suggests that, for the same architecture, reducing the patch size increases computational load and slows down inference speed.\n> \n> The paper says, “While reducing the patch size do not add parameters, it still leads to a significant reduction of running time, and larger memory usage.” That can sound as if smaller patches speed up inference. The table instead shows lower images/s for the same ViT when its patch shrinks, because the token sequence grows. Its “1.4× faster” comparison uses ViT-B/8 at 63 images/s against SimCLR v2 RN152w3+SK at 46 images/s; ViT-S itself falls from 1007 to 180 images/s when changing /16 to /8.\n> \n> I think that is the intended reading, but please correct me if I have misunderstood it.\n\n\n### Importance of Different Components\n\n\n\n![ablation](/content/papers/assets/00094ba4cbbff36270f09f383868ea59c1042a276138a50d6c17a8a45c4e8a17.jpg)\n\n\n\nThe authors conducted a series of experiments to assess the importance of various components:\n\n- **No Momentum (Row 2 in the table):**\n  When only the momentum teacher is removed and the other settings remain the same, both k-NN and linear accuracy fall to 0.1%.\n- **Operations to Prevent Collapse (e.g., SK):**\n  Row 9's SwAV has no momentum but uses the Sinkhorn-Knopp (SK) constraint and still reaches 71.8% linear accuracy. Rows 1 and 3 show little change when SK is added to DINO with momentum (76.1% versus 76.0%). Other settings also differ across rows, so the gap between rows 3 and 9 cannot be assigned entirely to momentum.\n- **Multi-crop and Cross-Entropy Loss:**\n  Relative to row 1, removing multi-crop in row 4 lowers linear accuracy from 76.1% to 72.5%; replacing cross-entropy with MSE in row 5 lowers it to 62.4%. These ablations test different components.\n- **Predictor:**\n  Adding a predictor to the student in row 6 gives 75.6%, 0.5 percentage points below row 1. BYOL relies on its predictor under a different method configuration.\n\n### Model Collapse Testing\n\n\n\n![collapse](/content/papers/assets/afbcfd295ead9f32cbe73b07f03e6b7bab0166f9d90440e92e6e0ac2da65c307.jpg)\n\n\n\nGenerally, the two most common types of collapse are:\n\n- The model outputs the same result for different inputs, collapsing into a uniform distribution.\n- One dimension overly dominates, leading to an imbalanced output.\n\nCentering subtracts the running center from teacher logits before softmax. It suppresses domination by a single dimension, but may encourage uniform outputs. A low teacher temperature sharpens the distribution and counterbalances that tendency.\n\nMathematically, the cross-entropy $H(P_t, P_s)$ can be decomposed into the teacher’s output entropy $h(P_t)$ and the KL divergence $D_{KL}(P_t \\| P_s)$:\n\n$$\nH(P_t, P_s) = h(P_t) + D_{KL}(P_t \\| P_s)\n$$\n\nA zero KL divergence means that teacher and student distributions agree. Collapse must also be assessed through output entropy and variation across samples.\n\n  The experiment shown in the figure indicates that if only one operation is used, KL converges to 0, while the teacher entropy $h(P_t)$ tends to different limits: 0 without centering, or $-\\log(1/K)=\\log K$ without sharpening. These correspond to one-dimension and uniform-output collapse, respectively. With both operations, the plotted entropy and KL avoid those two degenerate limits.\n\n## Conclusion\n\n\n\n![result](/content/papers/assets/1f996b170ca5dea4d5099d4d5d5653e91b61573cc8b0f26c7bac25f9c6f37c40.jpg)\n\n\n\nThrough extensive experimental analysis, the authors found that training ViT using self-supervised learning not only demonstrates exceptional performance in downstream tasks but can even rival convolutional neural networks (CNNs) specifically designed for vision tasks.\n\nThese experiments suggest that self-supervision gives ViT useful transfer features and emergent object-level attention.\n\nThe red regions are thresholded attention masks, not individual attention points or outputs from a trained segmentation head. In this visualization, the authors retain 60% of the attention mass and select the best head for the displayed object. The comparison illustrates emergent localization under that visualization procedure.\n\nThe results make self-supervised learning a promising way to train ViTs with less dependence on human labels. They support further exploration, rather than proving a universally optimal training method.\n","imageSizes":{"/content/papers/assets/b8cc1c529a1c65a2a6ce495941153b6b24def2bccc29907a0a5edf64b16570c9.jpg":{"width":1470,"height":526},"/content/papers/assets/c2d0ab75afb02750e17183e186f29b0ef6e184eda12673a31c96f199e338188f.jpg":{"width":664,"height":642},"/content/papers/assets/470e3d3a53b5913cc5c859a840bda8edd82b7430f1d787a5ae2f360e267d6440.jpg":{"width":820,"height":1080},"/content/papers/assets/00094ba4cbbff36270f09f383868ea59c1042a276138a50d6c17a8a45c4e8a17.jpg":{"width":736,"height":482},"/content/papers/assets/afbcfd295ead9f32cbe73b07f03e6b7bab0166f9d90440e92e6e0ac2da65c307.jpg":{"width":1224,"height":544},"/content/papers/assets/1f996b170ca5dea4d5099d4d5d5653e91b61573cc8b0f26c7bac25f9c6f37c40.jpg":{"width":1224,"height":620}}}
