Back to paper library
arXiv 2024CVPR 2024Face anti-spoofingPaper notes

CFPL-FAS

Class-Free Prompt Learning

Original paper

CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing

Paper authors

Ajian Liu · Shuai Xue · Jianwen Gan …

Show all 8 authors

Ajian Liu · Shuai Xue · Jianwen Gan · Jun Wan · Yanyan Liang · Jiankang Deng · Sergio Escalera · Zhen Lei

Publication
CVPR 2024Conference
First submitted to arXiv
arXiv:2403.14333
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{faceantispoofingcfplfas2024,
  title = {{CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing}},
  author = {Ajian Liu and Shuai Xue and Jianwen Gan and Jun Wan and Yanyan Liang and Jiankang Deng and Sergio Escalera and Zhen Lei},
  year = {2024},
  eprint = {2403.14333},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.14333}
}
On this page

This paper discusses a complex framework, but we can focus on three major keywords from the title to break it down:

  1. Class-Free
  2. Prompt Learning
  3. Face Anti-Spoofing (FAS)

By focusing on these three aspects, the paper becomes easier to understand.

Face Anti-Spoofing (FAS) is a critical defense mechanism in facial recognition systems, designed to prevent attackers from fooling the system using photos, videos, or other spoofing methods. With the focus on FAS, let's first explore some past approaches in the field.

Problem Definition

In earlier research, distinguishing real faces from fake ones was done by designing specific features such as texture, spectral information, and motion cues. However, these handcrafted features were limited in scope. This led researchers to move toward deep learning models like Convolutional Neural Networks (CNNs) to extract more advanced features of spoofed faces.

But as the saying goes, "As defenses evolve, so do attacks." Attackers began using high-quality images and videos to trick deep learning-based FAS systems.

To counter unknown types of attacks, the most popular recent strategy has been Domain Generalization, which aims to reduce the model's overfitting to the training data, allowing it to generalize better on unseen data.

With the rise of large language models (LLMs), multimodal training has also been applied to FAS, making it harder for attackers to spoof these systems using a variety of data types.


With this background in mind, let's dive into Prompt Learning as it applies to the problem.

Prompt learning gained popularity because training an LLM from scratch is expensive and impractical for most people. Instead, prompts are fed to pre-trained LLMs to generate useful features, which are then used to guide the training of our own models.

This is not a very new concept, and it has been widely known since CLIP. Subsequent research, such as BLIP-2, introduced the concept of Q-Former, aiming to transfer knowledge from large models using a small number of tokens to achieve cross-domain generalization.

Returning to this paper, the authors aimed to apply Prompt Learning in the FAS domain. However, the limited number of classes and the fact that pre-trained models aren’t specifically designed for FAS meant that even using prompt learning wouldn’t necessarily yield meaningful features, let alone help with generalization.

Thus, the problem was clear: the authors wanted to use prompt learning for FAS without relying on fixed class-specific prompts as classifier weights, while still achieving generalization. The final task still uses live/fake labels.

Now, let’s explore the solution.

Solution

Model Architecture

The above image shows the overall model architecture. However, looking at this without breaking it down is overwhelming. Let's break it into components.

Prompts Generation

FAS features are often hidden in the image and aren’t immediately apparent.

To capture these subtle features, the authors designed two sets of query tokens:

  1. Content Q-Former: A lightweight Transformer module whose query vectors interact with image-content features to generate content prompts.
  2. Style Q-Former: Another lightweight Transformer module whose queries interact with image-style statistics to generate style prompts.

This part of the architecture corresponds to the diagram below:

The query vectors are learnable, with 16 in each set. The authors compared several counts under ICM→O before choosing this setting.

The target features come from FAS images, extracted via CLIP's image encoder. However, instead of directly using the features from the image encoder, the authors take features from each layer of the encoder, calculate the mean and standard deviation, and use these to form style features vs\boldsymbol{v}_{s}:

vs=∑l=1LvslL,vsl=[μ(vl) ∣∣ σ(vl)],vs∈R1×2d\boldsymbol{v}_{s} = \frac{\sum_{l=1}^{L}\boldsymbol{v}^{l}_{s}}{L}, \quad \boldsymbol{v}^{l}_{s} = \left[\mu(\boldsymbol{v}^{l}) \,||\, \sigma(\boldsymbol{v}^{l})\right], \quad \boldsymbol{v}_{s} \in \mathbb{R}^{1 \times 2d}

Here, LL is the number of layers in the image encoder, and [⋅∣∣⋅]\left[\cdot || \cdot\right] indicates concatenation along the embedding dimension.

The content features vc\boldsymbol{v}_{c} are obtained directly from the image encoder's output, normalized as follows:

vc=vL−μ(vL)σ(vL),vc∈Rn×d\boldsymbol{v}_{c} = \frac{\boldsymbol{v}^{L} - \mu(\boldsymbol{v}^{L})}{\sigma(\boldsymbol{v}^{L})}, \quad \boldsymbol{v}_{c} \in \mathbb{R}^{n \times d}

Once the target features are defined, Content Q-Former queries the content features vc\boldsymbol{v}_{c}, and Style Q-Former queries the style features vs\boldsymbol{v}_{s}, yielding two sets of prompts.

During training, Diversified Style Prompt (DSP) also mixes means and standard deviations between different samples in the same batch so that SQF encounters more style variation; this step is disabled at test time.

Prompt Optimization

This section addresses the optimization of prompt features (the blue and green parts in the diagram).

The prompt features generated from the previous step cannot be directly used, as CLIP lacks domain-specific knowledge for FAS. To address this, the authors devised a binary classification task to help the Content Q-Former (CQF) understand FAS at a higher level.

The task is to predict whether a prompt–text feature pair matches. For instance, the authors use "a photo of a [CLASS] face", where [CLASS] is either live or fake. CLIP’s text tokenizer and embedding layer turn it into text-supervision features. “Class free” training still uses real/fake labels; the prompts are learned rather than fixed class-specific classifier weights.

To create positive and negative pairs, the authors draw on ALBEF for hard negative mining: according to prompt–text similarities, they sample mismatched text for each prompt and mismatched prompts for each text. The paper does not describe a separate memory bank storing prompt features.

After generating positive and negative pairs, each query passes through a two-class fully connected classifier; its logits are averaged and optimized with cross-entropy:

S=Embed(Tokenizer(T)),S∈RB×77×d\boldsymbol{S} = \texttt{Embed}(\texttt{Tokenizer}(\boldsymbol{T})), \quad \boldsymbol{S} \in \mathbb{R}^{B \times 77 \times d} S=Mean&Expand(S),S∈RB×N×d\boldsymbol{S} = \texttt{Mean\&Expand}(\boldsymbol{S}), \quad \boldsymbol{S} \in \mathbb{R}^{B \times N \times d} Rp=[P ∣∣ S]2,Rp∈RB×N×2d\boldsymbol{R}_{p} = \left[\boldsymbol{P} \,||\, \boldsymbol{S}\right]_{\texttt{2}}, \quad \boldsymbol{R}_{p} \in \mathbb{R}^{B \times N \times 2d} R=[Rp ∣∣ Rnprompt ∣∣ Rntext]0,R∈R3B×N×2d\boldsymbol{R} = \left[\boldsymbol{R}_{p} \,||\, \boldsymbol{R}^{prompt}_{n} \,||\, \boldsymbol{R}^{text}_{n}\right]_{\texttt{0}}, \quad \boldsymbol{R} \in \mathbb{R}^{3B \times N \times 2d} Lptm=∑i=13BH(yiptm,Mean(FCptm(Ri)))\mathcal{L}_{ptm} = \sum_{i=1}^{3B} \mathcal{H}\left(\boldsymbol{y}^{ptm}_{i}, \texttt{Mean}(\texttt{FC}^{ptm}(\boldsymbol{R}_{i}))\right)

Each query goes through a binary classifier, and its logits are averaged for the pair’s match score. Here H\mathcal{H} is cross-entropy and yptm\boldsymbol{y}^{ptm} is the matching label. This text supervision teaches CQF to distinguish matching from mismatching prompt–text pairs.


During training, three kinds of R\boldsymbol{R} are involved: Rp\boldsymbol{R}_{p} (positive pairs), Rnprompt\boldsymbol{R}^{prompt}_{n} (mismatched prompts), and Rntext\boldsymbol{R}^{text}_{n} (mismatched text). The PTM branch thus compares a matching prompt–text pair against both kinds of mismatch.

Class-Free Prompt Learning

Finally, let’s dive into the Class-Free Prompt aspect.

This part is actually simpler now that we’ve built up the prompt features and optimization mechanisms.

In some prompt-learning methods, text features act directly as the classifier weights for predefined categories. The authors argue that relying on those class descriptions can limit cross-domain generalization.

Instead, content and style prompts are generated dynamically for each image and turned into channel-modulation weights by the text encoder. The final task remains binary live/fake classification, and PTM also uses those labels. The model does not need text descriptions listing each attack type as fixed classifier weights.

This reduces the prompts’ dependence on predefined attack categories and lets the model learn how to adjust itself from examples.


The red-highlighted section is the Class-Free Prompt module.

The CQF and SQF prompts are separately fed into CLIP’s text encoder. Their text features are concatenated and passed through a sigmoid-gated network to produce one weight per channel, w\boldsymbol{w}. Each weight multiplies the corresponding visual feature from CLIP’s image encoder; a final fully connected classifier predicts live or fake and is optimized by cross-entropy.

The Prompt Modulation module is represented by the following formula:

w=δ(ge(t,W))=δ(W2σrelu(W1t)),\boldsymbol{w} = \delta (\texttt{g}_{e}(\boldsymbol{t}, \mathbf{W})) = \delta (\mathbf{W}_{2}\sigma_{\textrm{relu}}(\mathbf{W}_{1}\boldsymbol{t})), v~=[v~1,v~2,…,v~d],v~c=wc⋅vc,\boldsymbol{\tilde{v}} = \left[\boldsymbol{\tilde{v}}^{1}, \boldsymbol{\tilde{v}}^{2}, \dots, \boldsymbol{\tilde{v}}^{d} \right], \quad \boldsymbol{\tilde{v}}^{c} = \boldsymbol{w}^{c} \cdot \boldsymbol{v}^{c}, Lcls=∑i=1BH(yicls,FCcls(v~i)),v~∈RB×d\mathcal{L}_{cls} = \sum_{i=1}^{B} \mathcal{H}(\boldsymbol{y}^{cls}_{i}, \texttt{FC}^{cls}(\boldsymbol{\tilde{v}}_{i})), \quad \boldsymbol{\tilde{v}} \in \mathbb{R}^{B \times d}

Here σrelu\sigma_{\textrm{relu}} is ReLU; W1∈Rdr×2d\mathbf{W}_1 \in \mathbb{R}^{\frac{d}{r} \times 2d} and W2∈Rd×dr\mathbf{W}_2 \in \mathbb{R}^{d \times \frac{d}{r}} are learned fully connected weights, with reduction ratio r=16r=16. The channel weight wc\boldsymbol{w}^{c} multiplies visual feature vc\boldsymbol{v}^{c}; FCcls\texttt{FC}^{cls} then predicts the two classes from the adjusted visual features.

During training, the CLIP image encoder, CQF/SQF, gate, and classification heads are updated; the CLIP text encoder remains fixed. The learned prompts therefore guide an adapting image branch through a pretrained text representation.

Datasets, Protocols, and Evaluation Metrics

The authors follow two established protocols to evaluate their model’s generalization ability.

  • Protocol 1: Using four standard benchmarks: MSU-MFSD (M), CASIA-FASD (C), Idiap Replay-Attack (I), and OULU-NPU (O).
  • Protocol 2: Using RGB samples from CASIA-SURF (S), CASIA-SURF CeFA (C), and WMCA (W), which have a richer variety of attack types and environments.

In both protocols, each dataset is treated as a domain, and the leave-one-out approach is used to test generalization. Three metrics are applied:

  1. HTER (Half Total Error Rate): Average of False Rejection Rate (FRR) and False Acceptance Rate (FAR).
  2. AUC (Area Under Curve): Measures the ability to separate real and fake samples across decision thresholds.
  3. TPR at FPR=1%: Calculates the true positive rate at a fixed 1% false positive rate, useful for setting thresholds in real-world scenarios.

Implementation Details

  • Query Setup: Both content and style queries have a length of 16, with a dimension of 512 per query. CQF and SQF are set to a depth of 1.
  • Style Prompt Diversification: Activated with a 50% probability during training but not during testing.
  • Training Configuration: Batch size is 12, optimizer is Adam, weight decay is 0.05, and the second training stage has a minimum learning rate of 1e-6. Images are adjusted to 224×224, with random cropping and horizontal flipping. Training runs for 500 epochs.

Discussion

Experimental Results

Protocol 1: Among methods without CelebA-Spoof as extra training data, CFPL has the best HTER and AUC in all four target datasets in the table. The other methods do not report TPR@FPR=1%, so that metric cannot be ranked across them.

For instance, CFPL’s HTER on MSU-MFSD (M) is 3.09% (vs. 5.41%), on CASIA-FASD (C) it’s 2.56% (vs. 8.70%), on Replay-Attack (I) it’s 5.43% (vs. 10.62%), and on OULU-NPU (O) it’s 3.33% (vs. 8.86%).

The average HTER of 3.60% improves on the previous best listed result of 7.82%. With CelebA-Spoof added, CFPL* is better than FLIP-MCL* at 2.98% vs. 3.01% average HTER, but slightly behind ViTAF*-5-shot at 2.98% vs. 2.83%; the latter also uses five target-domain images in training.


Protocol 2: Without CelebA-Spoof, CFPL has lower HTER on WMCA (W) and CASIA-SURF CeFA (C) than the other methods listed, although its AUC and TPR@FPR=1% on C are not the best.

On CASIA-SURF (S), CFPL performs slightly worse than CLIP-V in HTER (8.77% vs. 8.32%), AUC (96.83% vs. 97.23%), and TPR@FPR=1% (53.34% vs. 57.28%). CLIP also reaches 60.75% TPR. The authors suggest that more obvious spoofing traces in CASIA-SURF let visual features generalize reasonably well. With CelebA-Spoof included, CFPL* averages 7.01% HTER, ahead of ViT* at 10.82% and FLIP-MCL* at 8.61%, while ViTAF*-5-shot reaches 6.83% under a different target-data condition.

Ablation Studies

  1. Effectiveness of Each Component

    • Baseline: This removes PTM, DSP, and PM while retaining CQF and SQF to generate prompts. Compared with CoOp’s freely learned vectors, these image-conditioned content and style prompts improve the table averages by 0.67 percentage points in HTER, 1.32 percentage points in AUC, and 7.88 percentage points in TPR@FPR=1%.
    • Adding text supervision further optimizes the model: HTER 7.50%, AUC 96.39%, and TPR@FPR=1% 54.78%.
    • Introducing style prompt diversification improves performance again: HTER falls by 0.42 percentage points, AUC rises by 0.40 percentage points, and TPR@FPR=1% rises by 2.83 percentage points.
    • Removing the designed gate worsens performance: HTER rises from 6.72% to 7.08%, AUC falls from 97.09% to 96.79%, and TPR@FPR=1% falls from 60.35% to 57.61%.
  2. Impact of CQF and SQF Structures

    • In the ICM→O experiment, simple two-layer bottlenecks struggle with the domain gap. Against the CoCoOp baseline, CQF alone and SQF alone reduce HTER by 1.68 and 1.96 percentage points, raise AUC by 1.38 and 1.48 percentage points, and raise TPR@FPR=1% by 13.26 and 26.67 percentage points, respectively.
    • The combination of CQF and SQF yields the best results, with HTER 3.33%, AUC 99.05%, and TPR@FPR=1% 90.06%.
  3. Impact of Query Length and Q-Former Depth

    • The authors experimented with query length ([8, 16, 32, 64]) and Q-Former depth ([1, 4, 8, 12]).
    • Key findings:
      1. A query length of 16 is close to the best: with depth one, 64 queries give the lowest HTER in the table, 3.30%, compared with 3.33% for 16, while other depths do not improve monotonically.
      2. Q-Former depth has a smaller, non-monotonic impact on performance in this experiment.

    Therefore, a query length of 16 and Q-Former depth of 1 are recommended.

Visual Analysis

Using attention maps, the authors compared the Baseline model with CFPL on selected examples. Baseline misclassifies these examples while CFPL classifies them correctly; the heatmaps can suggest where attention falls.

For example, in a Protocol 1 OCM→I example, the Baseline misclassified a real face and its heatmap leaned toward the background, while CFPL also attended near the face-background boundary and classified it correctly.

For Replay Attacks, the examples likewise show different regions of attention.

Conclusion

After reading this paper, we can see that the author's design approach is quite unique.

In this architecture, CLIP’s text encoder stays fixed, but its image encoder still participates in training. We want to take advantage of CLIP’s pretrained knowledge while adapting the image side to FAS.

Therefore, the entire design starts from CLIP’s Image Encoder, with the goal of using the Image Encoder to produce features that can classify FAS (Face Anti-Spoofing).

But CLIP doesn't know what FAS is, right? So, what should we do?

The authors introduced CQF and SQF. We can think of them as “plug-ins” that learn FAS prompts from image content and style; those prompts then pass through CLIP’s text encoder. They do not literally store a list of attack-class tokens.

CLIP’s text encoder has already been trained, so the idea of “combining” visual concepts is tempting. We might imagine words like “monitor” and “face” helping us think about Replay, or “paper” and “texture” helping us think about Print. But those are an analogy for the kind of knowledge we hope to reuse, not the paper’s literal learned token list or explicit attack-class prompts.

The specific imagined words are not important. What matters is letting the model learn image-conditioned prompts that bring useful FAS information through the text branch.

Once those prompts produce text features, the Prompt Modulation gate turns them into a weight for each visual channel. It multiplies those weights with CLIP image features and passes the result to a classifier. That is how the FAS cues enter the decision here, rather than a final text–image dot product.

Interesting, isn’t it?