Back to paper library
arXiv 2023ICCV 2023Face anti-spoofingPaper notes

FLIP

The Defense Spell

Original paper

FLIP: Cross-domain Face Anti-spoofing with Language Guidance

Paper authors

Koushik Srivatsan · Muzammal Naseer · Karthik Nandakumar

Publication
ICCV 2023Conference
First submitted to arXiv
arXiv:2309.16649
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{faceantispoofingflip2023,
  title = {{FLIP: Cross-domain Face Anti-spoofing with Language Guidance}},
  author = {Koushik Srivatsan and Muzammal Naseer and Karthik Nandakumar},
  year = {2023},
  eprint = {2309.16649},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.16649}
}
On this page

"This is real!"

With natural language as the noble witness, so says the model.

Problem Definition

At airports, smartphones, access control, and various verification points, the face has long become the most familiar identifier.

However, the anti-spoofing models (FAS) relied upon by these systems often reveal unexpected vulnerabilities when tested across devices, attack types, and environmental conditions:

  • Models perform well under familiar data distributions but degrade significantly once outside the training scenario;
  • Minor changes in camera resolution, lighting conditions, or even attack methods may cause the model to misclassify fake as real;

In the end, many methods assume access to samples from the target domain, but in practice, this is often a luxury. After all, no one knows where or how the next attack will come.

These same words have been heard for over a decade, yet the problem remains unsolved.

This forces us to reconsider a fundamental question:

Can a model still retain the perception of "liveness" in cross-domain scenarios with unknown types and lacking annotations?

Past attempts mostly revolved around three strategies seeking answers:

  • Unsupervised Domain Adaptation (UDA): relying on unlabeled target data to help generalize decision boundaries;
  • Few-shot Learning: quickly adjusting representations with a small number of labeled target samples;
  • Domain Generalization: trying to learn attack-irrelevant discriminative features from multiple source domains.

These approaches differ in their requirements: UDA needs unlabeled target data, few-shot adaptation needs some target labels, while domain generalization trains without target data but may rely on auxiliary information such as attack types or domain labels. Here, "zero-shot" means no target-domain training samples, not training without FAS labels. This makes zero-shot transfer an attractive yet difficult goal.

The authors of this paper propose a more radical idea:

  • What if we replace pretrained ViT with a multimodal model like CLIP? Would it naturally generalize better across domains?
  • If language itself can serve as a "style-neutral" descriptive aid, could it guide the model to learn semantic-level anti-spoofing intuition?
  • If images and text are not only paired but jointly encoded during training for "authenticity," could the model gain a deeper understanding of what is real?

These questions imply not just data structure issues but the semantic depth of representations. What the model needs to learn is not just features, but how to recognize the authenticity of a face from "descriptions."

And it all begins with a sentence:

"This is not a spoof face."


Problem Solution

Wait, you say you don’t know what CLIP is yet?

Let us briefly introduce:

Suppose we have a set of image-text pairs, for example, an image of a dog and the text "a cute puppy."

In one training batch, CLIP receives many such pairs. The image encoder—using ResNet or ViT—processes images to extract image features, while the text encoder—using a Transformer—processes text to extract text features.

Then, the model compares these features to ensure that the cosine similarity between correctly matched image-text pairs (e.g., dog image and "a cute puppy") is maximized, and between mismatched pairs (e.g., dog image and "an apple") is minimized.

Finally, it is trained on 400 million image-text pairs.

A very simple concept with impressive results, and the success of this architecture has opened a new era.

Model Architecture

The above figure illustrates the research architecture in this paper, corresponding as follows:

  • (a) FLIP-Vision: Simply uses CLIP’s vision encoder plus an MLP as the baseline architecture.
  • (b) FLIP-IT: Introduces textual descriptions as class correspondences, using cosine similarity as the classification criterion.
  • (c) FLIP-MCL: An evolved version under multimodal contrastive learning regularization, allowing the model to learn consistent cross-domain representations from more perspectives and self-validation.

This is a CLIP-based multimodal contrastive learning framework. Let’s see how CLIP’s knowledge can be transferred to the problem we want to solve.

FLIP-Vision

As mentioned earlier, CLIP's large-scale image-text pretraining supplies transferable representations for natural-image classification and, with task-specific designs, detection and segmentation.

But FAS must distinguish very subtle differences, such as paper reflections, screen textures, and recapture artifacts, so CLIP's features cannot simply be applied unchanged.

Thus, the authors propose FLIP-Vision (FLIP-V) as the baseline model, adopting a minimal adaptation strategy:

  • Retain CLIP’s image encoder;
  • Discard the text encoder and only use CLIP’s ViT backbone for feature extraction;
  • Take the class token output from the last Transformer block and feed it to a multi-layer perceptron (MLP) head for binary classification;
  • Train with standard Cross Entropy Loss to distinguish live from spoof faces.

This is visual fine-tuning initialized from image-text pretraining: FLIP-V updates the CLIP image encoder and MLP, using the class token before ImageProj, but does not use the text encoder during FAS fine-tuning.

During training, the authors use balanced image data from multiple source domains so the model learns discriminative cues across varying styles, attack types, and capture conditions.

This remains supervised FAS training with source live/spoof labels, without introducing language prompts during fine-tuning.

Therefore, FLIP-V essentially demonstrates one thing:

Even without language during fine-tuning, CLIP's visual backbone has a lower average HTER than the compared ImageNet-pretrained ViT under Protocol 1.

FLIP-Image-Text

Although the basic FLIP-Vision proves CLIP’s visual encoder has transfer potential on FAS tasks, it remains a “visual-only” classifier.

This design is still constrained within the “feature recognition” framework and does not realize CLIP’s original spirit: using language as a guide and decision reference for vision.

Hence, the authors introduce FLIP-IT.

The core idea of this version is to use natural language sentences as “semantic anchors,” allowing the model to directly compare the alignment between image features and textual descriptions during training, rather than learning solely from image classification boundaries.

This design no longer uses a traditional MLP head to classify images as real or fake. Instead, each image “selects the most similar description.”

The key logic is to represent each category (real/spoof) by a natural language sentence, then use CLIP’s text encoder to convert these sentences into vector representations z⃗r\vec{z}_r and z⃗s\vec{z}_s, and compute cosine similarity with the image vector x⃗\vec{x} as the prediction basis.

In other words, the model still classifies an image as real or spoof; similarity to the two class descriptions supplies its classification scores. FLIP-IT fine-tunes both the image and text encoders.

The authors hope that multiple descriptions of each class will guide the image representation toward cues relevant to authenticity. Whether a prompt set outperforms a single sentence depends on the tested domain.

In experiments, the authors designed six groups of prompts describing real and spoof classes, as shown below:

Prompt No.Real PromptsSpoof Prompts
P1This is an example of a real faceThis is an example of a spoof face
P2This is a bonafide faceThis is an example of an attack face
P3This is a real faceThis is not a real face
P4This is how a real face looks likeThis is how a spoof face looks like
P5A photo of a real faceA photo of a spoof face
P6This is not a spoof faceA printout shown to be a spoof face

These sentences represent different language strategies:

  • Some emphasize positive descriptions (e.g., P2: “This is a bonafide face”);
  • Some use negation structures (e.g., P3: “This is not a real face”);
  • Some resemble image captions (e.g., P5: “A photo of a real face”);
  • Some describe a live face by negating the spoof class and describe a specific attack for the spoof class, as in P6.

These form a set of “language context prompts.” The authors average the embeddings of six prompts per class (P=6) into zˉ\bar{z} for alignment with image features. Both encoders participate in FLIP-IT fine-tuning.

This gives the classification score several descriptions of the same class to draw on; whether the set is better than a single prompt must be judged from each test domain's results.

FLIP-Multimodal-Contrastive-Learning

We have seen how language transforms from “annotation” to “classifier” in FLIP-IT, enabling the model to align directly with semantic descriptions to learn authenticity. Yet this alignment is still based on single pairs and lacks robustness to variation and uncertainty.

Finally, the authors propose FLIP-MCL (FLIP-Multimodal Contrastive Learning).

The core idea of FLIP-MCL is to introduce contrastive learning into the multimodal framework, allowing the model not only to see correspondences but also to capture the “invariant” parts.

The overall training objective is expanded to three components:

  1. Classification loss LceL_{ce}: continuing FLIP-IT’s cosine similarity and cross entropy loss;
  2. Inter-image contrastive loss LsimCLRL_{simCLR}: enhancing consistent understanding of image content across different views;
  3. Image-text similarity consistency loss LmseL_{mse}: further promoting semantic stability between visual and textual contexts.

First, the simCLR part.

For each input image II, the authors randomly generate two augmented views Iv1I^{v_1} and Iv2I^{v_2}, such as by changing lighting, masking, or cropping. A view here need not come from a different physical camera angle.

Then, features are extracted by CLIP’s image encoder VV:

x⃗v1=V(Iv1),x⃗v2=V(Iv2)\vec{x}^{v_1} = \mathcal{V}(I^{v_1}), \quad \vec{x}^{v_2} = \mathcal{V}(I^{v_2})

Next, a nonlinear projection head H\mathcal{H} maps these to the contrastive space:

h⃗1=H(x⃗v1),h⃗2=H(x⃗v2)\vec{h}_1 = \mathcal{H}(\vec{x}^{v_1}), \quad \vec{h}_2 = \mathcal{H}(\vec{x}^{v_2})

Finally, the simCLR loss LsimCLRL_{simCLR} brings the two views of one image closer in the contrastive space, while using other images' views in the batch as negatives.


Next, the MSE part.

On the language side, FLIP-MCL looks for consistency across different sentences. The authors randomly sample two prompts z⃗v1\vec{z}^{v_1} and z⃗v2\vec{z}^{v_2} from the image's ground-truth class and pair them with its two augmented views. Spoof images use spoof prompts, and live images use live prompts.

For pairs (x⃗v1,z⃗v1)(\vec{x}^{v_1}, \vec{z}^{v_1}) and (x⃗v2,z⃗v2)(\vec{x}^{v_2}, \vec{z}^{v_2}), cosine similarities are computed, and an MSE loss forces these similarities to be close:

Lmse=(sim(x⃗v1,z⃗v1)−sim(x⃗v2,z⃗v2))2L_{mse} = \left(\text{sim}(\vec{x}^{v_1}, \vec{z}^{v_1}) - \text{sim}(\vec{x}^{v_2}, \vec{z}^{v_2})\right)^2

This essentially tells the model:

Whichever augmented view you see, and whichever prompt you read from its class, the two image-text similarities should be close.


In the end, the authors combine the three objectives into the total loss:

Lmcl=Lce+LsimCLR+LmseL_{mcl} = L_{ce} + L_{simCLR} + L_{mse}

At inference, the model still compares the image with the real and spoof class text representations by cosine similarity, as FLIP-IT does.

Discussion

This study evaluates performance using three cross-domain testing protocols:

  • Protocol 1 employs four classic datasets: MSU-MFSD, CASIA-MFSD, Idiap Replay Attack, and OULU-NPU. Following a leave-one-domain-out setting, one dataset is held out as the target domain while the others serve as training sources.
  • Protocol 2 uses three large-scale face anti-spoofing datasets: WMCA, CASIA-CeFA, and CASIA-SURF. The leave-one-out strategy is similarly applied to validate generalization across devices and complex environmental variations.
  • Protocol 3 simulates single-source to single-target scenarios, selecting from the MCIO four datasets to form 12 domain-pair tests, modeling practical cases where data sources are limited.

Evaluation metrics include:

  • Half-Total Error Rate (HTER, lower is better)
  • Area Under the ROC Curve (AUC, higher is better)
  • True Positive Rate (TPR) at False Positive Rate (FPR) = 1% (higher is better)

The authors repeat each experiment with five seeds and report the mean. All three protocols also include CelebA-Spoof as supplementary training data, so Protocol 3's “single source” refers to its benchmark-domain pairing, not literally training on only one dataset. Some listed baselines also differ in target-data access.

Protocol 1: MCIO Four Domains

From the table, three main trends emerge:

First, FLIP-V (visual fine-tuning) has a lower HTER than the zero-shot ViT in the C, I, and O target domains, but a higher one in M. Its average HTER is 3.48%, below ViT's 6.00%. CLIP initialization with full visual fine-tuning is competitive under this protocol, though it does not win on every target.

Second, FLIP-IT (language supervision) improves results on C, I, and O over FLIP-V, with large TPR@FPR=1% gains. This supports using language prompts to guide the decision, while individual metrics and domains still need separate comparison.

Finally, FLIP-MCL has the lowest average HTER, 3.01%. Looking at individual domains, it leads on O; FLIP-IT leads on C and I, while zero-shot ViT leads on M. FLIP-MCL also has lower HTER than the listed five-shot ViTAF results on C and O, but its AUC and TPR are not highest in every column.

The gap on M relative to zero-shot ViT is especially clear: 4.95% versus 1.58% HTER. The paper's misclassified examples include real faces rejected as attacks; the authors suggest that resolution and lighting changes may contribute.

Overall, Protocol 1 supports FLIP-MCL's average HTER advantage on these four test domains, with exceptions for individual domains and metrics.

Protocol 2: WCS Three Domains

Protocol 2 extends the experiment to WMCA (W), CASIA-CeFA (C), and CASIA-SURF (S), with more subjects and varied attacks and sensors. CeFA and SURF each exceed a thousand subjects, but WMCA has 72; the model uses RGB images rather than fused multisensor input.

Relative to zero-shot ViT, FLIP-V has HTER lower by 1.85, 0.24, and 0.87 percentage points on W, C, and S; its mean falls from 10.82% to 9.83%.

FLIP-IT raises TPR@FPR=1% over FLIP-V by 8.88 and 5.47 percentage points on W and C. On CW → S, however, its HTER, AUC, and TPR all worsen.

FLIP-MCL has better HTER and AUC than the other listed zero-shot methods on all three targets. Its TPR leads on W and S, while on C it is slightly below FLIP-IT (59.00% versus 59.40%). Against zero-shot ViT, its TPR is higher by 10.25, 11.41, and 8.01 percentage points on W, C, and S.

Compared with Protocol 1, these datasets contain more subjects and environmental variation. FLIP-MCL achieves a lower mean HTER under these three cross-dataset settings, though the table does not isolate the effects of subjects, devices, or other shifts.

Protocol 3: Single Source → Single Target

Protocol 3 simulates an extreme but common practical case: training data from a single source domain with limited resources and direct inference on an unseen target domain.

Using MSU-MFSD (M), CASIA-MFSD (C), Idiap Replay (I), and OULU-NPU (O), 12 cross-domain tasks were designed to test generalization under data scarcity and domain shift pressure.

Results show:

First, FLIP-V averages 7.12% HTER across the 12 pairs, below the previously listed methods with all 12 results. Other methods still do better on individual pairs.

FLIP-MCL averages 4.84% HTER, 8.36 percentage points below CDFTN-L's 13.20%. Comparing the two rows in Table 4:

  • C → O falls from 29.90% to 3.91%, 25.99 percentage points lower.
  • I → O falls from 29.90% to 4.22%, 25.68 percentage points lower; M → C falls from 8.80% to 0.19%, 8.61 percentage points lower.
  • C → I, M → I, and O → M favor CDFTN-L, though FLIP-MCL keeps the lowest 12-pair mean in the table.

The lower 12-pair mean supports FLIP-MCL in this single-source benchmark setup. The table alone cannot separate the contributions of simCLR, prompt averaging, and MSE.

Overall, Protocol 3 shows an average advantage without target-domain training samples; reliability in actual deployment would need separate evaluation.

Pretraining Strategies

To clarify the impact of initialization on cross-domain generalization, the authors compared various ViT initialization methods: training from scratch (no pretraining), self-supervised pretraining (e.g., BeIT), ImageNet pretrained models, and multimodal pretrained weights from CLIP.

CLIP initialization has the lowest mean HTER and the lowest HTER on C, I, and O. On M, ImageNet initialization is better (1.58% versus 3.79%). This supports CLIP as an effective starting point, without isolating image-text semantics as the sole cause.

Prompt Design

In language-supervised frameworks (FLIP-IT, FLIP-MCL), the wording of textual prompts significantly affects classification performance.

The table illustrates differences in performance across various cross-domain scenarios depending on prompt design, showing no single universally optimal prompt exists, and reliance on semantic phrasing varies by scenario style.

The authors also compare single prompts with the ensemble: the ensemble's mean HTER is 3.06%, better than most single prompts but worse than P6's 2.94%. A single prompt also leads on each of C, I, and O. Prompt choice matters.

These results show that wording affects performance under each tested setting.

Loss Weight Analysis

FLIP-MCL combines three losses: supervised Cross Entropy, self-supervised simCLR, and image-text similarity consistency (MSE). The MSE prompt pair is sampled from the image's ground-truth class, so that term is label-conditioned. To assess each loss's contribution, the authors varied their weights, as shown above.

Removing either self-supervised loss raises mean HTER in this test: 3.15% without MSE and 3.47% without simCLR, versus 3.01% with equal weights. Giving the self-supervised terms much larger weights (1:5:5) also raises HTER to 3.67%. The comparison supports combining the tested objectives at suitable weights, but does not isolate the internal cause of each change.

The three-loss design therefore needs suitable weights; the label-conditioned MSE term does not make the whole training procedure self-supervised.

Conclusion

While most prior works remained stuck on stacking visual features and cycling datasets, the authors chose a new path:

Starting from multimodal foundational models like CLIP and introducing language as a semantic anchor for judgment.

Building on this, they incorporate a visual-language contrastive learning framework to establish an anti-spoofing strategy centered on semantic alignment.

This approach yields several key insights:

  1. Representations learned by multimodal pretrained models have strong cross-domain potential.

    Even fine-tuning only the image encoder markedly improves generalization, showing that these models embed a semantic space structure far more transferable than traditional ImageNet models.

  2. Language is no longer just an auxiliary annotation but a learnable classification criterion.

    Through sentence-defined class contexts, models learn “what kind of reality this is,” rather than merely judging “does this look real” from pixels.

  3. Multimodal contrastive learning provides a more general regularization strategy.

    In scenarios with scarce annotations and limited sample diversity, it helps models align diverse visual contexts and semantic explanations, enhancing stability under extreme domain shifts.

Although FLIP-MCL introduces extra computation by using a language encoder during training, experiments prove this cost is justified by significant performance gains and cross-scenario adaptability.

In future tasks involving more complex, multi-source data, language may become the key medium for deep model generalization.