Back to paper library
arXiv 2022CVPR 2023Multimodal learningPaper notes

FLIP

Fast CLIP

Original paper

Scaling Language-Image Pre-training via Masking

Paper authors

Yanghao Li · Haoqi Fan · Ronghang Hu · Christoph Feichtenhofer · Kaiming He

Publication
CVPR 2023Conference
First submitted to arXiv
arXiv:2212.00794
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityflip2022,
  title = {{Scaling Language-Image Pre-training via Masking}},
  author = {Yanghao Li and Haoqi Fan and Ronghang Hu and Christoph Feichtenhofer and Kaiming He},
  year = {2022},
  eprint = {2212.00794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00794v2}
}
On this page

This paper, proposed by Meta AI, introduces a new approach to improve the training efficiency of CLIP.

Problem Definition

In early 2021, OpenAI introduced the CLIP architecture, which sparked a wave of contrastive learning applications in the industry.

While CLIP is simple and effective, training it is highly resource-intensive.

In the original CLIP paper, the authors built WIT-400M, a dataset of 400 million image-text pairs. The largest RN50x64 model required 592 V100 GPUs for 18 days; that was not a basic CLIP model. ViT-L/14 also took 12 days on 256 V100s.

For independent engineers and researchers, obtaining such computing resources is far from achievable. Meta AI recognized this problem and proposed FLIP (Fast Language-Image Pre-training) as a solution.

Solution

Model Architecture

FLIP keeps CLIP's two-tower image-text contrastive architecture, but during training it adds random image-patch masking. Its encoder implementation also differs from original CLIP in some details.

The idea is straightforward: since the image encoder in the model uses a ViT structure, the authors drew inspiration from the MAE training strategy. This strategy shows that a good model can be trained using only a portion of the image data.

The authors tried it. In the paper's same-data experiments, masking 50% or 75% of image patches takes about 0.50 or 0.33 times the unmasked training time, respectively; accuracy depends on the setting. At a similar target accuracy, Figure 1 separately reports roughly a 3.7-fold speedup.

And just when we thought we could relax and happily finish the paper... (nope, there’s more to dive into! Here we go again!)

Masking Strategy

For image masking, FLIP adopts the same approach as MAE. The image is divided into non-overlapping patches, and a large portion of these patches is randomly masked. The masking ratio can be set to 50% or 75%.

Since masked patches never enter the image encoder, the visual-encoding part can use roughly 2× or 4× larger batches at similar memory use. Other model and contrastive-batch costs remain, so total memory is not guaranteed to stay fixed. Larger batches are also one reason accuracy can improve.

While CLIP includes both images and text, FLIP's main experiments mask only the image input. Text masking is tested in an ablation but dropped: the text tower is smaller and shorter, so the total speed gain is limited.

During inference, the model processes the full, unmasked image without requiring any adjustments to the encoder.

Loss Function

Although FLIP borrows the masking mechanism from MAE, it does not use a reconstruction loss. Based on ablation experiments, the authors found that reconstruction loss does not improve zero-shot transfer performance. By removing the decoder and reconstruction head, they achieved better speed and simplicity.

FLIP relies entirely on contrastive loss. In this setup:

  • Positive pairs are corresponding image-text pairs.
  • Negative pairs are drawn from other samples within the same batch.

The image and text encoder outputs are projected into the same embedding space and normalized. Their cosine similarity is scaled by a learned temperature. The following InfoNCE formula illustrates the image-to-text direction:

L=−1N∑i=1Nlog⁡exp⁡(sim(vi,ti)/τ)∑j=1Nexp⁡(sim(vi,tj)/τ)L = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\text{sim}(v_i, t_i) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(v_i, t_j) / \tau)}

where NN is the batch size, viv_i and tit_i are the embedding vectors for the ithi^{th} image and its corresponding text, and sim(vi,tj)\text{sim}(v_i, t_j) denotes the cosine similarity between the image and text embeddings. τ\tau is a learnable temperature parameter.

Cosine similarity is computed as:

sim(vi,tj)=vi⊤tj∥vi∥∥tj∥\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\| \|t_j\|}

Actual CLIP/FLIP training also computes the text-to-image direction and averages the two losses. The objective draws matched images and texts together and pushes other within-batch pairings apart, though semantically matching "negatives" can occur in real data.

Discussion

What improvements enable FLIP to significantly accelerate CLIP’s training? In this section, we summarize the key findings from the ablation studies conducted by the authors.

Ablation Studies

Table 1 uses ViT-L/16 after 6.4 epochs on LAION-400M and reports ImageNet-1K zero-shot accuracy. Most timings use 256 TPU-v3 cores; the measured 0.33× time for 75% masking is about a threefold wall-clock speedup here, rather than a universal fourfold speedup.

  1. Masking Ratio

    • The model is trained on LAION-400M and evaluated on the ImageNet-1K validation set for zero-shot accuracy.
    • Table (a) examines masking ratios while adjusting batches for similar image-encoding memory: no masking at batch 16k gives 68.6%; 50% masking at batch 32k gives 69.6%, 1.0 percentage point higher; 75% masking at batch 64k gives 68.2%, 0.4 point lower.
    • Training time:
      • With 50% masking, training time is reduced to 50% of the original time.
      • With 75% masking, it is reduced to 33% of the original time.
  2. Batch Size

    • Table (b) explores the impact of batch size on model accuracy:
      • Increasing batch size consistently improves accuracy.
      • Even with a batch size of 16k, the model with 50% masking achieves 68.5% accuracy, comparable to the unmasked model’s 68.6%.
      • The authors suggest masking may have a regularizing effect that partly offsets lost information; the table does not separately measure overfitting.
      • However, 75% masking combined with a fixed batch size causes performance degradation due to information loss.
    • Scaling the batch along with the mask ratio can keep visual-encoding memory similar. But Table 1(b)'s 50% mask, 64k batch takes about twice the memory of the comparable settings in (a), so memory limits remain.
  3. Text Masking

    • Table (c) shows that randomly masking 50% of text reduces accuracy by 2.2 percentage points:
      • Text data is more information-dense than image data, making large-scale text masking less effective.
      • Prioritizing the masking of padding tokens reduces the accuracy drop to 0.4 percentage points.
    • Text masking provides limited speed benefits: with the image unmasked, the text encoder costs about 4.4% as much computation as the visual encoder. The main experiments therefore leave text intact.
  4. Inference Unmasking

    • Table (d) reveals that ignoring the difference between masked and unmasked distributions during inference yields good results, even in zero-shot settings:
      • With the 75%-mask-trained model, masking at inference changes 68.2% to 60.9%, a 7.3-percentage-point drop; the corresponding 50%-trained model falls by 3.2 points.
      • Using ensemble methods that combine multiple masked views reduces the performance gap but still lags behind using the full image.
  5. Unmasked Tuning

    • Table (e) shows that continuing on the pretraining dataset with 0% masking for 0.32 additional epoch raises the 75%-masked model from 68.2% to 69.5%, up 1.3 points; the 50%-masked model rises from 69.6% to 70.1%.
      • This unmasked tuning is consistent with closing the masked-pretraining versus intact-inference distribution gap. It does not use ImageNet class labels.
  6. Reconstruction Loss

    • Table (f) explores the impact of adding a reconstruction loss:
      • The reconstruction head follows the design in MAE, using a small decoder to reconstruct normalized image pixels. The reconstruction loss is combined with the contrastive loss.
      • In Table 1(f), adding reconstruction lowers zero-shot accuracy from 69.6% to 69.4% at 50% masking and from 68.2% to 67.9% at 75%; the authors also report a small fine-tuning decline. Other reconstruction weights or settings were not exhaustively ruled out.
    • To simplify the system and improve performance, the authors ultimately omit the reconstruction head and loss.

Comparison with CLIP

To demonstrate the effectiveness of FLIP, the authors compare it against several CLIP baselines, showing that FLIP is a strong alternative to CLIP. The baselines used include:

  • Original CLIP: Trained on the private WIT-400M dataset.
  • OpenCLIP: A faithful reproduction of CLIP, trained on the public LAION-400M dataset.
  • Authors’ CLIP reproduction: Also trained on LAION-400M, this version helps analyze methodological differences.

Zero-Shot Transfer on ImageNet

The table above compares zero-shot accuracy on ImageNet-1K.

  • Authors' CLIP reproduction achieves slightly higher accuracy than OpenCLIP, both trained on LAION-400M.
  • FLIP: With batch 64k, 50% masking, and unmasked tuning, its ViT-L/14 reaches 74.6%, 1.8 percentage points above OpenCLIP's 72.8% and 1.5 points above the authors' 73.1% CLIP reproduction on LAION-400M. The L/14 result also receives adjustment after L/16 pretraining, rather than starting with an identical L/14 schedule.
  • Original CLIP's ViT-L/14 scores 75.3%, 0.7 point higher than FLIP, but used private WIT-400M. This cross-dataset comparison does not isolate masking.

Zero-Shot Classification on Additional Datasets

This table compares zero-shot classification performance across several datasets.

  • WIT-trained models excel in tasks like Aircraft, Country211, and SST2, while LAION-trained models perform better on Birdsnap, SUN397, and Cars.
  • In most listed tasks, FLIP outperforms the two LAION-400M CLIP baselines. Green highlights indicate the best among the LAION entries, not necessarily all WIT rows. There are exceptions: on GTSRB, FLIP scores 41.4% versus 50.2% for the authors' CLIP reproduction; on PCam, 50.3% is below the two LAION comparators' 51.1% and 53.1%.

Zero-Shot Retrieval

This table shows image-text retrieval results on Flickr30k and COCO.

  • Compared with both LAION-400M CLIP baselines, FLIP leads all twelve listed retrieval metrics. Against the original CLIP authors' 224-pixel reevaluation, FLIP leads most but not all: Flickr30k text-retrieval R@5/R@10 are 98.5/99.6 versus 99.1/99.8. Original CLIP also has a 336-pixel row, which is a different input setting.
  • These are two directions of zero-shot retrieval: text retrieval searches captions from an image; image retrieval searches images from text. The WIT versus LAION data difference remains when comparing with original CLIP.

Model Scaling

With the efficiency gains from FLIP, the authors explore how to push beyond CLIP’s limits by scaling along three axes:

  1. Model Scaling

    • The ViT-L image encoder is replaced with ViT-H, which has approximately 2x the parameters, with a corresponding increase in the text encoder.
  2. Data Scaling

    • The pre-training dataset is expanded from LAION-400M to LAION-2B.
    • To isolate the effect of data volume from training duration, the total sampled data is kept at 12.8B (equivalent to 32 epochs on 400M or 6.4 epochs on 2B data).
  3. Schedule Scaling

    • The sampled data is increased to 25.6B (64 epochs on LAION-400M).
    • In single-axis tests the other two settings stay fixed; the last two table rows test joint model-and-data and then three-axis scaling.

Results of the Scaling Experiments are shown in the figure below:

  1. Model Scaling (Fig. a): Accuracy steadily improves throughout training, although the gap narrows towards the end.
  2. Data Scaling (Fig. b): Early performance is close to baseline, while later accuracy improves. Total sampled pairs stay at 12.8B, so the curve does not require more sampled training examples; it uses more diverse source data.
  3. Schedule Scaling (Fig. c): Doubling sampled pairs to 25.6B yields limited late zero-shot improvement and does not exceed the 12.8B baseline at the end of the plotted comparison. The dashed line is a hypothetical horizontal rescaling of the long-schedule curve, not an actual shorter run.

Transferability

The table above details the impact of different scaling strategies on downstream tasks:

  • Both model scaling and data scaling beat the baseline on the listed metrics. ImageNet-1K zero-shot accuracy rises from 74.3% to 75.5% and 75.8%, respectively.
  • Data scaling keeps 12.8B sampled pairs, supporting a gain from greater data variety under that condition rather than from seeing more total examples.

The second-to-last row shows joint model-and-data scaling exceeding either single-axis change in the listed metrics. On ImageNet-1K, model scaling adds 1.2 percentage points (74.3→75.5), data scaling 1.5 points (74.3→75.8), and joint scaling 3.3 points (to 77.6). The authors infer that larger models may benefit from more varied data.

The last row scales model, data and schedule: ImageNet-1K zero-shot accuracy reaches 78.8%, above the cited OpenCLIP LAION-2B result of 78.0%. It is not an improvement in every transfer column: COCO/nocaps caption CIDEr fall from 130.4/92.6 in the prior row to 130.2/91.2.

OpenCLIP used 32B sampled pairs and FLIP 25.6B. Combining those counts with an assumed 2.5× FLIP speed ratio on comparable hardware, the authors estimate about 3,360 GPU-days saved. This is a planning calculation, not a direct measurement of both training runs on identical hardware.

Conclusion

Through a series of ablation experiments and comparative analyses, the authors demonstrate the effectiveness of FLIP. When training CLIP models on large-scale datasets, FLIP significantly accelerates training speed and achieves better performance on multiple zero-shot tasks.

Time is money, and we believe no one would have a problem with money, right?

Do you want to build a CLIP model? FLIP is undoubtedly a cost-effective choice.