FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models
- Publication
- AAAI 2026Conference
- First submitted to arXiv
- arXiv:2505.09415
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofingfaceshield2025,
title = {{FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models}},
author = {Hongyang Wang and Yichen Shi and Zhuofu Tao and Yuhao Gao and Liepiao Zhang and Xun Lin and Jun Feng and Xiaochen Yuan and Zitong Yu and Xiaochun Cao},
year = {2025},
eprint = {2505.09415},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2505.09415}
}
Do you still remember the dataset paper SHIELD we read not long ago?
This time, let's take a look at the related follow-up applications.
Problem Definition
Face Anti-Spoofing (FAS) is an indispensable security mechanism in facial recognition systems that prevents spoofing attacks such as photos, videos, and masks from misleading the model. Over the past decade, the research community has proposed a large number of methods addressing this problem, which can be roughly divided into two main directions:
The first type consists of vision models built on CNNs or Vision Transformers, sometimes supplemented with cues such as reflected light, depth maps, rPPG, and image gradients. They treat the task as binary classification between live and fake faces; these methods can perform well on familiar data but often generalize poorly to unseen scenes or new attack types.
The second type consists of newer vision-language methods, which train models using paired images and text, with CLIP as a representative example. These methods use semantic signals from language descriptions to assist judgments and reduce reliance on spurious features.
However, VLM methods still have two major unresolved issues:
- First, they lack explainability and cannot answer why a face is fake;
- Second, they cannot localize forged regions on the fake face nor handle classification and reasoning tasks simultaneously.
Even worse, both visual models and VLM methods suffer from severe scarcity of training data. Most FAS datasets only contain labels and images, lacking descriptive annotations that support reasoning tasks, which directly limits the language ability and generalization potential of the models.
What about Multimodal Large Language Models (MLLMs)?
Although general-purpose MLLMs such as GPT-4V and LLaVA have demonstrated strong capabilities in tasks like remote sensing, medical imaging, and deepfake detection, they perform only moderately on FAS tasks. The main reason is that current language models lack specific knowledge about face spoofing attacks, and off-the-shelf vision encoders struggle to detect subtle texture differences between real and fake faces.
Some prior works attempted instruction tuning to build FAS models with descriptive capabilities and integrated multiple datasets to improve semantic recognition, but such methods still cannot simultaneously handle classification, reasoning, and localization tasks. Real-world attacks are often composite in nature, and these limitations make it difficult for existing models to be practical.
Thus, the question arises:
Can we build a model that truly understands the suspicious aspects of a fake face, can explain why, and even point out where the deception starts?
Proposed Solution
To enable a multimodal language model to learn to detect fake faces, just using text and images is insufficient.
It requires task-specific, semantically complete, and structurally coherent training data, and even needs to understand “why this face is suspicious” at multiple levels. The authors believe the key starting point is the data.
To this end, the authors first designed the concept of their dataset as shown below:
FaceShield uses datasets built for two training stages:
- FaceShield-pre10K: for pretraining, enabling the model to acquire basic descriptive and comprehension abilities;
- FaceShield-sft45K: for instruction fine-tuning (SFT), strengthening the model’s concrete performance across multiple tasks.
The research team integrated popular FAS datasets including WMCA, PADISI, and SiW-Mv2, unifying their labels into 12 categories:
Bonafide, Fakehead, Print, Glasses, Replay, Paper mask, Flexible mask, Rigid mask, Partial Eye, Partial Mouth, Makeup, Tattoo
Not only did they keep image-level labels, but also provided precise bounding boxes and coordinates for locally forged regions, resulting in 12,091 images with category labels and 3,139 images with forged region annotations.
After collecting images and labels, the next step was to help the language model understand this information.
The authors used Bunny-Llama-3-8B-V as the generation assistant, guiding the model with system prompts to produce descriptive sentences and generating corresponding QA pairs according to different task types.
For FaceShield-pre10K, the process was relatively straightforward: the model generated descriptive text based on images, which were then filtered by CLIP similarity (below 15% removed) to discard low-similarity image–text pairs.
For FaceShield-sft45K, the process was more rigorous, covering four types of task instructions along with multi-level filtering and enhancement strategies:
- The MLLM first outputs multiple QA pairs, which are filtered manually and by keywords to remove errors;
- Then LLaMA3 is used to linguistically enhance the remaining QA seed data, improving sentence diversity and dialogue fluency.
The four tasks are illustrated as follows:
- Coarse-grained Classification: recognizing whether the face is real or fake;
- Fine-grained Classification: identifying the specific attack type of the fake face;
- Reasoning: verbally describing the judgment basis from image features;
- Attack Localization: detecting forged face regions and marking attack coordinates.
Model Architecture
Training a multimodal large language model (MLLM) tailored for face anti-spoofing (FAS) tasks has two core objectives:
- Enable the visual encoder to better distinguish subtle differences between real and fake faces;
- Enable the language model to answer different task questions using semantic prompts.
However, if training directly on RGB images paired with QA, as shown in (a), the model faces a challenge:
- In RGB space, real and fake faces look too similar.
This makes it difficult for the model to stably learn reliable discriminative features, limiting generalization.
To solve this problem, the authors introduced two key modules:
- SAVP (Spoof-Aware Vision Perception): enhancing the model’s perception of spoof details via visual preprocessing;
- PVTM (Prompt-Guided Vision Token Masking): dynamically adjusting the model’s attention focus based on language prompts.
Together, these two modules aim to make the FAS MLLM more sensitive to attack cues and better able to respond to the four tasks.
Spoof-Aware Vision Perception (SAVP)
As illustrated in (b), the core idea of SAVP is that appearance in RGB space alone is insufficient to effectively distinguish live from spoof faces, but combining image texture and gradient information can amplify these subtle differences.
Specifically, for each input image, SAVP computes three additional local features, which are combined into a three-channel input:
- LBP (Local Binary Pattern): reflects local texture variation
- Gray (Grayscale): removes color interference, emphasizing luminance structure
- HOG (Histogram of Oriented Gradients): captures edge orientation and gradient energy distribution
These image features are concatenated as three channels and fed into the visual encoder to extract spoof-aware feature representation , which is then concatenated with the original RGB features as the total visual input:
This design provides the model with two types of visual cognition: one from appearance, the other from texture and structure.
Prompt-Guided Vision Token Masking (PVTM)
Even with rich visual features extracted, if the model cannot focus on task-relevant regions, it can still be distracted.
PVTM offers a mechanism to dynamically select visual tokens as follows:
- Pool the language prompt tokens into , then compute their cosine similarity with each visual token :
- Convert to weight ranking scores via softmax:
Then keep the top ranked tokens and randomly mask the remaining tokens with probability ; the experiments set and . Lower-ranked tokens are not all discarded.
This enables the model’s visual attention to vary with prompts for tasks such as classification, reasoning, and localization, enhancing task adaptability and robustness against noise.
Model Workflow
The overall architecture shown in (c) flows as follows:
- Convert input images into two sets of features: and ;
- Merge and apply PVTM to select the most important visual tokens;
- Project the selected tokens to align the visual input with the language model, yielding ; the following formula summarizes this step without showing PVTM explicitly:
- Input together with the prompt into the language model, producing task prediction :
Two-Stage Training
FaceShield adopts a two-stage training strategy:
-
Stage 1: Pretraining
Using the previously constructed FaceShield-pre10K, this stage aligns features extracted by the visual encoder with language prompts, establishing correspondence channels between vision and language.
- The vision encoder comes from a pretrained model; only the Projector and PVTM modules are fine-tuned;
- Training follows continual pretraining on existing weights;
- Loss function is cross-entropy, optimizing next-token prediction as the language objective;
- Pretraining lasts for one epoch, focusing on rapid embedding space alignment rather than strengthening semantic generation.
-
Stage 2: Instruction Fine-tuning (SFT)
This critical adaptation stage uses FaceShield-sft45K to fine-tune the model to answer four specific task queries.
Fine-tuned modules include:
- LoRA layers (lightweight adapters inside the language model)
- Projector (vision-language bridging module)
LoRA (Low-Rank Adaptation) efficiently adjusts the large language model by inserting LoRA submodules in each Transformer block, with parameters:
- Hidden size: 128
- Scaling factor: 256
The loss remains cross-entropy with next-token prediction, but emphasis shifts to language performance and task responsiveness.
Discussion
Intra-dataset Evaluation
On the combined test subsets of the three source datasets (W/S/P), FaceShield has a lower HTER than the traditional FAS methods and general-purpose MLLMs listed in the table:
-
Traditional FAS methods
- The table lists ResNet, PatchNet, and CoOp, whose architectures and training methods differ.
- Their HTERs range from 1.27% to 2.32%, compared with FaceShield's 0.53%.
-
Zero-shot capability of general-purpose MLLMs
- Such models are not fine-tuned for FAS tasks, relying solely on prior language/vision knowledge for inference.
- LLaVA, Qwen-VL, MiniGPT4, and Bunny all have higher HTERs than FaceShield in this table; GPT-4V is not included in this comparison.
-
Finetuned MLLM baseline: Bunny
- The authors fine-tuned the open-source MLLM “Bunny” on the same data (SFT) as a strict baseline.
- FaceShield's HTER is 0.53%, 0.99 percentage points below the fine-tuned Bunny's 1.52%.
Cross-dataset Evaluation
This table presents cross-dataset generalization results.
Each experiment trains on two of the three datasets (e.g., S + P) and tests on the excluded one (e.g., W), denoted as S&P → W. There are three such permutations corresponding to three domains.
FaceShield’s HTER is lower than the listed traditional methods in all three settings. For S&P → W, the table reports 5.71% HTER, while the paper's prose says 5.72%. The table contains no MLLM baseline.
Ablation Studies
-
Effectiveness of Instruction Dataset
If the instruction dataset is effective, can it improve other models as well?
The authors applied the same dataset for pretraining and finetuning on open-source MLLMs like LLaVA and Bunny.
Results (shown below) improve coarse classification, fine classification, and reasoning for both tested models; for example, LLaVA's coarse HTER falls from 27.76% to 1.74%, while Bunny's falls from 17.87% to 1.06%. The figure does not report a mean improvement across tasks.
This supports the usefulness of the dataset for these two MLLMs.
-
Effectiveness of SAVP (Spoof-Aware Vision Perception)
Comparing the top two rows: baseline (RGB only) versus RGB + SAVP.
The table shows improvement across all four tasks in this setting. In attack localization, AP@40 rises from 92.30% to 97.09%, and AP@50 from 89.71% to 95.21%.
This supports the use of local texture and gradient cues for the tested recognition and localization tasks.
-
Effectiveness of PVTM (Prompt-Guided Vision Token Masking)
Referring to the last row of the same table:
PVTM improves coarse classification, fine classification, and localization in this ablation. Reasoning declines slightly: ACC falls from 99.41% to 99.29%, and HTER rises from 0.48% to 0.57%. The authors suggest masking may remove needed context, but the ablation does not isolate that cause.
Visualization Analysis
If a model claims a face is fake, where is it “looking” to make this claim?
The authors offer a visualization to explore this question.
Figure 10(a) shows token importance scores—how strongly visual tokens respond to language prompts—across three categories, including a bona fide face.
High token responses often fall near the face, though responses also appear elsewhere. The similarity map alone does not show which cues determined the final answer.
These examples show different visual-token responses to different prompts, but cannot by themselves verify the stated reason for a decision.
Figure 10(b) compares RGB and SAV tokens for “attack region detection”:
-
Left: an eye mask example:
- RGB token importance is highly dispersed with no clear focus;
- SAV tokens show higher responses around the eye-mask attack region;
-
Right: an example with an eye attack region and a hand holding an object:
- RGB and SAV tokens respond around these regions, offering complementary visual cues; the figure does not separately measure hand-occlusion detection accuracy.
Conclusion
This paper extends traditional binary classification into a multi-task framework including coarse classification, fine classification, semantic reasoning, and localization. The model can answer whether a face is real or fake, produce a written reason, and predict a box around an attack region; whether its stated reason actually drove the judgment needs separate verification.
Such capability moves the tested system beyond a single “yes or no” output toward task-specific text and location predictions. Figure 10 helps inspect token responses, while Table 6 shows that the prompt-guided mask slightly worsens the reasoning metric.
The ability to say “why it is fake” and “where the fake is” represents current expectations and major research trends.