InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
- Publication
- ACM Multimedia 2025Conference
- First submitted to arXiv
- arXiv:2507.12060
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofinginstructflip2025,
title = {{InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing}},
author = {Kun-Hsiang Lin and Yu-Wen Tseng and Kang-Yang Huang and Jhih-Ciang Wu and Wen-Huang Cheng},
year = {2025},
eprint = {2507.12060},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2507.12060}
}
Face recognition technology is a mirror.
It reflects the progress of artificial intelligence in "seeing" human appearances, while also exposing the fragility of our definition of "reality."
Problem Definition
This technology has moved from the periphery of biometrics into daily life, replacing passwords, unlocking devices, and even deciding who can cross borders.
However, when a high-resolution photo, a looping video, or even a perfectly shaped silicone mask can fool the system, we must admit: this mirror is not reliable.
Face Anti-Spoofing (FAS) was born to address this. Researchers started from binary classification models, gradually adding attention mechanisms, generative adversarial networks, and auxiliary learning modules, trying to expose the illusion of fake faces.
Next, Domain Adaptation (DA) and Domain Generalization (DG) emerged to address cross-device and cross-scene problems. DA can require target-domain data, while leave-one-domain-out DG evaluation may require repeated training for different held-out domains. Each new evaluation setting can therefore come with a cost the industry cannot ignore.
Until 2023, the emergence of FLIP brought new imagination.
It first introduced the language modality from CLIP, attempting to incorporate the "semantics of faces" into the detection system, using language to describe the authenticity of the scene. This approach opened the door for cross-modal FAS, but language only played a supportive role; semantics had not yet become core.
Soon after, CFPL-FAS continued this path, using BLIP-2's Q-Former to build image-conditioned content and style prompts. Its prompts do not themselves become a fixed set of attack-class text weights, though live/fake labels still supervise training.
However, its reported domain-generalization evaluations still involve separate leave-one-domain-out training runs. The method uses structured prompts for classification rather than free-form attack descriptions.
In 2025, I-FAS further explored language by using spoof-aware captions and instruction tuning to produce judgments with descriptions. Its Protocol 1 uses leave-one-domain-out training; its separate Protocol 2 trains once on CelebA-Spoof and tests on eleven unseen datasets. These settings answer different questions about generalization.
In other words, these studies—from FLIP to CFPL to I-FAS—while continuously expanding the semantic boundary of FAS tasks, all left sighs by the same wall:
Models cannot understand the world with a unified language, nor maintain consistent reasoning across diverse environments.
So, do we really need more training and bigger models to climb over this wall? Is there untapped power in language? Can we build a model that can understand language, see through illusions, and learn the whole picture at once?
Problem Solution
One of the greatest challenges models face in FAS tasks comes from the feature distribution gap caused by different datasets, scenes, and devices.
Traditional solutions often cast a wide net: train on several domains and hold out one at a time to assess transfer. Changing the held-out target entails another training run, adding to the cost.
To solve this, InstructFLIP proposes a unified generalization framework with the core idea:
Instead of letting the model blindly guess the attack and environment behind each face, directly give it clear language instructions.
This strategy is built on a key dataset: CelebA-Spoof.
CelebA-Spoof provides attack-type and scene-condition annotations, allowing explicit prompts within the range of those labels. The model can learn from these annotated conditions in one training run.
To effectively cover the semantic space, InstructFLIP’s instructions are clearly divided into two categories:
-
Content Prompts: Asking "What type of attack is this image showing?"
Options include photo, screen, flat mask, partial occlusion, 3D mask, and a real-face class—ten spoof classes plus real, eleven options in total. For example:
“Choose the correct option for the following question: Which spoof type is in this image?”
-
Style Prompts: Asking "What is the shooting environment of this image?"
Options cover lighting conditions (normal/backlight/dark/strong light), scenes (indoor/outdoor), camera quality (low/medium/high). For example:
“What is the illumination condition in this image?”
“What is the environment of this image?”
“What is the camera quality in this image?”
This language structure turns content and style labels into separate instructions, supervising the model to learn attack types and capture conditions.
Model Architecture
The above figure shows the entire InstructFLIP architecture, which can be divided into three core modules:
- (a) Content Branch
- (b) Style Branch
- (c) Feature Fusion and Judgment Module (Fusion + Classifier & Cue Generator)
Below, we analyze the figure step-by-step following the data flow.
Content Branch
The Content Branch is responsible for learning semantic representations directly related to attack types, aiming to enable the model to judge which spoof attack category an input image belongs to, or whether it is a real face.
The process starts from the image , which is passed through the visual encoder to obtain deep features representing the content semantics of the image. Then, it enters the language-driven key module Q-Former, bridging language instructions and image semantics.
Specifically, the authors design a set of content prompts in the form of multiple-choice questions, for example:
"Choose the correct option for the following question: Which spoof type is in this image?"
Options range from real face to various spoof types (photo, screen replay, 2D mask, 3D mask, etc., totaling 11 types). These prompts are converted into instruction embedding vectors , concatenated with a set of learnable query vectors , serving as initial input to Q-Former:
Inside Q-Former, multiple layers of attention modules are stacked:
- Multi-Head Self-Attention (MSA): Updates internal relations between queries and semantic instructions:
- Cross-Modal Attention (MCA): Interacts the hidden state with image content features ; the paper's Equation (3) still denotes this input by :
This process enables the query vectors to "correspond" to the semantic targets the instruction aims to find, e.g., "Does this face have suspicious attack features?" or "Does it show a specific spoof pattern?"
The final query representation, combined with the original instruction embeddings, forms a soft prompt fed into the frozen Large Language Model (LLM) layer for semantic-guided prediction, producing content branch prediction .
The loss function uses standard cross-entropy:
where is the ground truth class label. This supervises the content branch with attack-type and real-face labels; whether it filters out background interference requires experimental evidence.
Style Branch
The Style Branch acts more like a background observer, primarily tasked with understanding "how this face was captured."
Domain shifts in FAS tasks often come not from attack method changes but from differences in "environment and device." For example, a real face taken at dusk with low resolution may look more like an attack sample than a brightly lit, clear fake face.
This explains the role of the style branch: lighting, environment, and camera quality supervise it to learn variations distinct from attack type.
Structurally symmetrical to the content branch, the style branch also receives a set of style prompts such as:
- “What is the illumination condition in this image?”
- “What is the camera quality in this image?”
- “Is this photo taken indoors or outdoors?”
These instructions are encoded into embedding vectors , concatenated with learnable query vectors , then fed into the Q-Former module. This allows the model to learn: "Under what environmental conditions are which visual features unreliable?"
Regarding feature design, the style branch uses a different construction approach from the content branch:
- First, it extracts multi-layer intermediate outputs from the visual encoder: .
- For each layer, the mean and standard deviation are calculated to form a statistical representation:
- Finally, all layers are combined into a multi-scale style feature:
This design is inspired by Adaptive Instance Normalization (AdaIN) used in style transfer tasks.
The style branch prediction is also semantically aligned by language guidance and optimized with cross-entropy loss for recognition accuracy:
The style feature is not directly inserted into the final fusion, but its query vectors do enter that fusion and the final judgment.
Feature Fusion and Decision Module
After the Content Branch and Style Branch each complete the parsing of task semantics and scene semantics, InstructFLIP does not simply concatenate their outputs. Instead, it designs a semantically guided fusion module that integrates knowledge from both channels as the basis for the final decision.
The key operation in this module comes from the attention mechanism within the Q-Former architecture. The authors first concatenate the learned query vectors from the Content Branch and Style Branch () into a fusion query:
Then, this query is further interacted with the content features via cross-modal attention to obtain the final fused representation :
Notably, the style feature is not inserted into this fusion, although the style queries are. The authors argue that feeding domain-specific style features directly into fusion could weaken generalization.
In other words, the style branch shapes the fused queries without passing its raw style feature into the final attention step.
The fused representation is then split into two parts:
- : a leading token fed into the spoofing classifier to predict whether the sample is spoof or real.
- : the remaining tokens passed into the cue generator to produce a cue map; this map has no human-annotated attack-region supervision.
Multi-Task Learning
To simultaneously optimize language understanding, visual reasoning, and interpretable output, the authors use a composite loss function:
- / : classification losses from the Content and Style Branches (semantic alignment).
- : final spoof/real classification loss, using standard cross-entropy between prediction and ground truth.
- : loss for the cue map learning, inspired by one-class training, defined as:
where is the difference between the predicted cue map and a class-derived target map, and is the threshold switching between the two penalty expressions. Spoof targets are all white and bona fide targets all black; the paper does not use human-annotated attack regions here. As printed, the two expressions have unequal limits at , so implementation should be checked before reproducing the loss.
This design provides an auxiliary "white map = spoof, black map = bona fide" signal. It does not make the cue map a region-supervised localization prediction.
Implementation Details
InstructFLIP uses CelebA-Spoof (CA) as the sole training source.
This dataset provides eleven content options—ten attack types plus the bona fide class—and environment annotations such as:
- Illumination conditions
- Shooting environment
- Camera quality
These semantic annotations construct rich content prompts and style prompts, enabling the model to learn features driven by natural language instructions.
Unlike prior leave-one-out or one-to-one training strategies, InstructFLIP adopts a train once → test on multiple datasets approach, conducting generalization evaluations on seven public FAS datasets:
- MSU-MFSD (M)
- CASIA-FASD (C)
- Replay-Attack (I)
- OULU-NPU (O)
- WMCA (W)
- CASIA-CeFA (C)
- CASIA-SURF (S)
This protocol represents a stricter generalization scenario, meaning the model requires no adaptation to target domains and directly handles unknown attack styles and style distributions.
Model implementation details include:
-
Input processing: All face images aligned via MTCNN and resized to RGB inputs.
-
Visual backbone: CLIP’s ViT-B/16 model as feature extractor .
-
Language model (LLM): FLAN-T5 base, kept frozen during training.
-
Training strategy: AdamW optimizer with OneCycleLR scheduler. Initial learning rate , peak learning rate , weight decay .
-
Training setup: Single NVIDIA RTX 4090, batch size 24, total 20 epochs.
-
Loss weights:
- (Content Branch)
- (Style Branch)
- (Final Classifier)
- (Cue Generator)
These are the paper's training settings. Deployment cost depends on the actual model variant; the supplement also reports an InstructFLIP⁻ variant without the frozen LLM.
For evaluation metrics, the authors follow prior literature conventions using three classic indicators:
- HTER (Half Total Error Rate)
- AUC (Area Under ROC Curve)
- TPR@FPR (True Positive Rate at specific False Positive Rate)
To ensure stability and credibility, each model is trained and tested with five different random seeds, reporting average performance.
Discussion
To evaluate InstructFLIP’s true strength under a unified setting, the authors include current representative FAS methods for comparison, covering diverse design philosophies and technical routes:
- SSDG: uses style separation to enhance domain-invariant features.
- ViT: pure vision transformer architecture; here tested in zero-shot mode excluding ViTAF few-shot for fair comparison.
- SAFAS: employs self-adaptive filters to boost generalization.
- FLIP: pioneering research introducing CLIP for cross-modal FAS.
- CFPL: further incorporates BLIP2 and Q-Former to expand language prompt use in FAS tasks.
Due to lack of public code, BUDoPT, DiffFAS, and FGPL are excluded from implementation comparison.
All other methods are re-implemented and retrained under the authors’ unified experimental protocol to ensure evaluation consistency.
Experimental results are shown below:
-
HTER (Error Rate): InstructFLIP beats the listed comparisons on six of seven datasets; CASIA-CeFA is the exception, where its 8.49% HTER is higher than CFPL's 5.50%. Against CFPL, the other six HTERs are:
- MSU-MFSD (M): 8.45% → 5.52%
- CASIA-FASD (C): 2.77% → 1.47%
- Replay-Attack (I): 10.00% → 9.12%
- OULU-NPU (O): 17.21% → 14.33%
- WMCA (W): 26.85% → 19.51%
- CASIA-SURF (S): 42.29% → 30.33%
-
AUC (Area Under Curve): higher than CFPL on six datasets, but 96.81% on CASIA-CeFA versus CFPL's 98.74%.
-
TPR@FPR=1% (True Positive Rate at low false positive rate): higher than CFPL on six datasets, but 70.17% on CASIA-CeFA versus CFPL's 78.72%.
CASIA-CeFA trails CFPL on all three metrics. The paper suggests cultural and environmental differences as possible causes; these factors were not ablated.
Module Contribution Analysis
In the ablation study, the authors first evaluate the contribution of the three main modules:
- Content Branch (CB)
- Style Branch (SB)
- Cue Generator (Cue)
The baseline model uses "no language prompts, image-only input," serving as a fair comparison starting point.
- Adding CB alone lowers mean HTER from 21.96% to 14.25% and raises AUC from 82.88% to 90.23%.
- Adding SB alone gives 16.25% mean HTER and raises TPR@FPR=1% from 26.89% to 41.92%; CB and SB together raise that TPR to 60.84%.
- Adding Cue to CB and SB lowers mean HTER from 13.25% to 12.68% and raises TPR@FPR=1% from 60.84% to 65.23%.
Together, these ablations support contributions to mean classification metrics under the unified evaluation.
Impact of the LLM Module
To clarify whether performance gains originate from the semantic capabilities of the large language model (LLM) or from the structured annotations provided by the meta-domain, the authors designed a variant called InstructFLIP†:
- Retains the Content Branch (CB) and Style Branch (SB) but replaces the LLM with a lightweight classifier.
Results show:
- Without the frozen LLM, InstructFLIP† reaches mean HTER/AUC of 14.39%/89.97%, better than CFPL's 16.15%/88.65%; its TPR@FPR=1% is 48.33%, below CFPL's 53.98%.
- With the frozen LLM, the full model reaches 12.68% HTER, 93.68% AUC, and 65.23% TPR@FPR=1%. This comparison supports a contribution from the LLM in this architecture, but does not isolate "language reasoning" as the cause.
In other words, structured annotations and the frozen LLM both affect this set of model variants, and the benefit depends on the metric.
Granularity of Spoof Type Classification
In this experiment, the authors evaluated how finely the attack types are labeled. Print, replay, and 2D masks each grow from one to three subcategories, while the bona fide class and 3D masks remain single categories.
Moving from one to three subcategories for each of those attack types lowers mean HTER from 19.11% to 12.68% and raises AUC from 87.14% to 93.68%. Yet the middle two-subcategory setting lowers TPR@FPR=1% from 39.13% to 38.30%, so every extra split does not improve every metric.
This supports finer attack labels in this dataset and model setting, without showing that the same taxonomy suits every task.
Diversity of Style Prompts
Finally, the authors examined the impact of different numbers of style prompts on performance. They incrementally added:
- Style 1: Illumination condition
- Style 2: Shooting environment
- Style 3: Camera quality
Across the three tested combinations, mean HTER is 20.65%, 19.25%, and 12.68%, while TPR@FPR=1% is 37.48%, 45.08%, and 65.23%. Adding all three style prompts helps in this setting, though the table does not directly measure whether the features have become disentangled.
Qualitative Analysis
Beyond quantitative metrics, the authors conducted qualitative observations to analyze InstructFLIP’s actual inference behavior, covering successful cases, failure cases, and comparisons with open-source vision-language models.
To ensure fairness and semantic completeness, four sets of prompts were designed for each sample:
- Content: Detect attack type
- Style1: Illumination condition
- Style2: Shooting environment
- Style3: Camera quality
This prompt design, shown in the table below, gives the model separate questions about content and capture conditions.
Successful Case Analysis
As shown in (a), a live face sample was correctly classified as genuine with a reported fake score of 0.001 for this example.
Similarly, in (b), a spoof image (tablet screen replay) was correctly identified as fake with a fake score of 0.998.
In these successes, we observe:
- Semantic outputs predicted by the LLM (e.g., spoof type, illumination, environment, image quality) can be compared with the labels shown for these selected cases.
- The cue maps respond to parts of the images, but were trained with all-white/all-black class targets rather than annotated spoof boundaries.
- These examples show the output format on selected cross-domain samples.
Failure Case Analysis
Despite strong overall performance, InstructFLIP still faces misclassification risks in some scenarios.
Failure cases are categorized into:
-
A spoof image classified as real
As in (c), the model incorrectly recognized a poster spoof sample as a real face. The figure also shows:
- A low fake score of 0.055 for this poster sample.
- An illumination prediction of Backlight rather than the listed Normal label. The example does not isolate material gloss or luminance gradients as the cause.
-
A real image classified as spoof
As in (d), the model misclassified a real face as screen replay, with a fake score of 0.851. Some other predicted attributes match the labels and others do not.
These cases highlight the need to assess attribute prediction and live/spoof decisions separately.
Comparison with Open-Source VLM Models
To illustrate prompt responses, the authors chose two models for a small comparison:
- InstructBLIP: a general instruction-following vision-language model.
- GPT-4o: a general multimodal model.
The table shows one fake and one live sample under content and style prompts. In these two examples, GPT-4o declines or gives incomplete answers to some questions, while InstructBLIP makes some attribute errors. The table does not estimate how often either model behaves this way in general.
For these selected examples, InstructFLIP's answers align more closely with the displayed labels; broader stability and reasoning superiority require a larger controlled evaluation.
Conclusion
InstructFLIP continues CFPL’s research line on semantic-driven FAS. Although superficially similar in architecture—both use Q-Former with multimodal prompt design—InstructFLIP makes significant improvements in three key aspects:
First, through a meta-domain training strategy, it evaluates one model trained on CelebA-Spoof across seven target datasets rather than retraining for each leave-one-out fold. That reduces training redundancy in this evaluation; deployment costs were not measured.
Second, it uses separate content and style branches for attack-type and capture-condition supervision before fusing their learned queries. The ablations support a benefit from both branches in this setting.
Finally, InstructFLIP introduces query fusion and a cue map generator. The full reported model still includes the frozen LLM; the supplement separately evaluates a variant without it, at 0.030 versus 0.046 seconds per sample on one RTX 4090. Phone latency and model-switching cost require separate measurement. The cue map is trained without attack-region annotations.
From an engineering standpoint, InstructFLIP shows one way to use language as a supervision signal while testing a lighter variant. Prompts can be an interface between models and task specifications, provided their extra cost and the faithfulness of the resulting explanations are checked in the intended setting.