Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models
- Publication
- AAAI 2025Conference
- First submitted to arXiv
- arXiv:2501.01720
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofingifas2025,
title = {{Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models}},
author = {Guosheng Zhang and Keyao Wang and Haixiao Yue and Ajian Liu and Gang Zhang and Kun Yao and Errui Ding and Jingdong Wang},
year = {2025},
eprint = {2501.01720},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2501.01720}
}
Face anti-spoofing is commonly treated as a binary classification task: Real or Spoof.
However, if we ask "Why?", most models cannot provide an answer.
In fact, they do not even allow you to ask "Why?".
Problem Definition
Existing face anti-spoofing models are mostly trained on a single objective — classifying input images as either real or fake.
This logic has been used for years, but frequently fails in scenarios such as:
- Large style gaps between test scenes and training data
- New spoofing techniques (e.g., high-definition replay, 3D masks)
- User groups with cultural, device, or geographic differences
Researchers have tried various approaches to strengthen generalization ability, including:
- Domain Adaptation (DA): Exposing the model to target domain data for feature alignment
- Domain Generalization (DG): Learning stable feature distributions from multiple source domains
- Incremental Learning (IL): Introducing new data domains while preserving prior knowledge
Yet, regardless of the method, the essence remains extracting "liveness features" from images — features that the models themselves mostly cannot interpret.
This is the core issue:
We train a system that can judge but cannot explain.
It can tell you "this is fake," but cannot specify "where it is fake" or "why it is fake."
We let the model see the image but never taught it how to tell a reasonable story.
Ironically, human recognition is precisely based on storytelling.
Solution
"Is this real?"
For face anti-spoofing models, this question should not only have an answer but also an explanation.
The I-FAS framework is built on this belief, translating the FAS task into an "image question answering" problem.
To make the model answer like a human, three aspects need to be addressed: language construction, feature alignment, and training strategy.
Spoof-aware Captioning and Filtering (SCF)
Most general captioning models only focus on facial expressions, clothing, or background, but ignore key spoofing clues.
As illustrated below:
The traditional captioner output is denoted as , while after fine-tuning by the authors, the spoof-aware captioner is .
As shown, traditional captions tend to be vague, whereas the fine-tuned spoof-aware captioner mentions an attack medium such as screen or paper in these examples.
The SCF process is as follows:
- Aggregate 12 public datasets into , labeled with real or fake samples .
- Build a dictionary of spoof keywords , categorized by attack types.
- Use a general captioner to generate initial descriptions for all fake images.
- Filter out captions that do not contain corresponding keywords, keeping a subset that clearly reflects attack features.
- Fine-tune on to obtain the spoof-aware captioner .
- Finally, generate captions for all real images to form training data with semantic differences.
This strategy equips each data sample not only with a label but also with a semantic description for training. Whether that description explains the model's actual decision requires separate validation.
Multimodal Instruction Tuning with GAC
After completing the preparation of training data, we move on to the model training stage.
The overall model architecture is shown above. I-FAS consists of three main components:
- Visual Encoder : Extracts features from input image .
- Cross-modal Connector : Maps into the token space of the language model.
- Language Model : Automatically answers questions and generates natural language descriptions based on visual features and textual prompts.
The visual encoder and language model in the diagram remain frozen; the GAC connects visual features to the language model.
Globally Aware Connector (GAC)
To achieve more comprehensive and detailed visual semantics, the authors propose the GAC module, which aims to transmit multi-level global features to the language model. The detailed process is as follows:
-
Extract (i.e., cls tokens) from each layer of the visual encoder, forming .
-
Linearly project these into and concatenate with learnable query vectors :
-
Then, undergoes multi-head self-attention (MSA) and multi-head cross-attention (MCA) combined with regional features :
This means using as the main query to attend to image features and find liveness-specific representations.
Typically, shallow features help capture texture details (e.g., moiré patterns, blur), while deep features tend to abstract semantics; GAC integrates these perspectives into the language output. Thus, the input to the language model is not a single feature, but a cross-modal representation that fuses visual depth, spatial, and semantic hierarchies.
Lopsided Language Model Loss (L-LM Loss)
Once the model learns to "speak," the next question is:
Should it speak the "result" first or the "reason" first?
The authors reformulate the FAS task into a single-turn question-answer format, with training data composed of three parts:
where:
-
is a fixed instruction sentence: "Is this photo of a real person?"
-
is a two-segment answer format:
Here, is the model’s real/fake decision (“Yes” or “No”), and is a semantic explanation generated by the captioner ( or ) to justify the judgment.
Using traditional LM loss, the model treats the entire as continuous text for autoregressive learning.
This leads to two issues:
- Unclear semantic focus: The model may overly mimic the style of the caption at the sentence end, neglecting key judgment tokens.
- Noise hampers convergence: Explanations often contain detailed noise, offering limited help to the core FAS decision and slowing the model’s convergence on the main task.
To give the correct judgment more weight during training while still learning an explanation, the authors adopt an asymmetrically weighted language model loss called Lopsided LM Loss (L-LM Loss):
where:
- is the token-level loss for the judgment part (Yes/No)
- is the loss for the explanation part
- is a hyperparameter controlling the training focus
Changing adjusts the relative weight of judgment and interpretation tokens in one training objective. The paper does not describe a staged curriculum that first trains judgment and then gradually adds explanations. At inference, the probability of answering "Yes" is used as the live-face score.
The reported HTER/AUC and training-loss curves show better results and faster, less variable loss reduction with L-LM Loss in the tested setting.
Discussion
Cross-Domain Generalization Evaluation
First, Protocol 1 uses four classic datasets in a leave-one-domain-out test.
As shown in the table, I-FAS has the lowest HTER in all four tests, averaging 1.33%. Its AUC is highest in three of the four: for O&C&M→I, its 98.48% trails DiVT-M's 99.29%. These results support cross-domain performance under the tested protocol; the table alone cannot attribute the whole difference to natural-language supervision.
Protocol 2 pushes the experiment to an extreme: training only on CelebA-Spoof as a single source, while testing on 11 unseen datasets differing in attack types and sensing devices.
Experimental results:
Under this setting, I-FAS averages 93.71% AUC, 5.81 percentage points above FLIP's 87.90% and 10.89 points above ViTAF's 82.82%. For SURF-3DMask and SIW-M-V2, both HTER and AUC beat the listed comparisons. On HKBU-MARs-V1+, AUC edges out FLIP, but HTER is 18.64% versus FLIP's lower 17.25%.
Module Contributions
To verify the contribution of each module, the authors performed separate ablations, removing one of the three core components from the full model at a time. Results are shown above:
- Removing SCF also reverts the target answer to Yes/No only. HTER rises by 3.17 percentage points and AUC falls by 3.30 points; this ablation changes both the caption supervision and the explanation output.
- Removing GAC, while keeping the token count with learnable queries, raises HTER by 3.54 points and lowers AUC by 2.93 points. This supports the contribution of multilevel global features in this setting.
- Replacing L-LM Loss with standard LM Loss raises HTER from 11.30% to 12.06% and lowers AUC from 93.71% to 92.81%. The loss curve is also more variable, though these results do not directly measure overfitting.
Caption Type Analysis
Does text diversity necessarily imply better generalization?
The authors conducted relevant experiments. The table compares various textual supervision formats, illustrating that semantic content in captions is key:
- In I-FAS, judgment tokens alone yield 14.47% HTER and 90.41% AUC. Adding generic captions yields 14.23% and 90.22%, so the two metrics do not improve together. For FLIP, replacing templates with spoof-aware descriptions changes 18.73%/87.90% to 17.30%/89.15%.
- With spoof-aware captions that mention attack clues, the listed mean AUC rises to 93.71%, compared with 90.22% for the corresponding generic-caption setting.
- Replacing spoof-aware captions with generic captions () decreases performance, indicating that specificity of information matters more than corpus diversity.
L-LM Loss Hyperparameter Study
Among the tested values, about 0.4 corresponds to the highest mean AUC and lowest mean HTER. The right-hand plot separately compares training loss with and without L-LM Loss:
- If is too high, interpretation has less weight in the objective, and the plotted cross-domain metrics worsen.
- If is too low, interpretation has more weight; the authors suggest caption noise may hinder learning, and the plotted is also worse than 0.4.
This comparison supports balancing the two losses. The curves alone do not show that the model has learned a structured reasoning process.
Explanation Capability Visualization
Earlier we compared spoof-aware captions with generic captions; the former mention media such as paper, tablet, or screen in the selected examples, providing more task-focused text supervision.
Finally, the figure below shows selected answers on SIW-M-V2 and Rose-Youtu after training on the single CelebA-Spoof source. They contain judgment and description, illustrating the output format. Whether the model actually used the stated clues for its judgment requires validation beyond these examples.
Conclusion
When models can only learn to answer "yes" or "no," we can only expect a confidence score output, but remain blind to why and based on which clues the decision was made.
This classification mindset, while helpful for system development and evaluation, traps models in a black-box framework, struggling to cope with cross-domain generalization, evolving attack types, and changing deployment scenarios.
This study attempts to redefine the output format of this task by introducing natural language question answering and caption-assisted supervision, enabling models not only to "answer true or false" but also to "explain their reasoning."
This shift in supervision syntax reflects a broader trend in computer vision:
From single-label to structured semantic descriptions.
Experimental results show that this semantic guidance brings substantial benefits. In Protocol 1's multi-source test, I-FAS has the lowest HTER in all four folds; in Protocol 2's single-source test, it leads the reported averages, with some per-dataset exceptions.
This suggests that language can aid learning in these tested settings, though the faithfulness of the generated explanations still needs verification. This rewrite driven by language is not merely the process of models learning to speak, but an opportunity to rethink the boundaries of the FAS task.
The moment the model begins to speak, perhaps not only FAS but
All classification tasks will eventually face a different destiny.