MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing
- Publication
- IJCAI 2022Conference
- First submitted to arXiv
- arXiv:2304.07549
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofingmavit2023,
title = {{MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing}},
author = {Ajian Liu and Yanyan Liang},
year = {2023},
eprint = {2304.07549},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2304.07549}
}
The initial purpose of Face Anti-Spoofing (FAS) technology was to block attacks that deceive face recognition systems using photos, videos, or 3D masks.
These attacks, known as Presentation Attacks (PAs), have become a critical concern for the security of biometric systems as face recognition technology gains widespread use in applications such as financial payments, smartphone unlocking, and self-service immigration checks.
Problem Definition
The earliest FAS methods primarily relied on RGB visual information to detect anomalies.
However, the cat-and-mouse game between attacks and defenses has never stopped. Attacks have become more realistic, and sensor devices have become increasingly diverse. We began to introduce infrared, depth, and thermal imaging modalities, aiming to piece together a "multi-spectral truth."
The model architectures evolved accordingly: from halfway fusion in intermediate feature concatenation to late fusion at the decision stage, each strategy attempted to more robustly reconstruct the "boundary between truth and falsehood."
However, new problems have quietly emerged.
While halfway fusion can enhance the interactive learning of different modal features, its model design often assumes consistent input modalities during both training and testing phases. If a sensor is absent, the overall prediction could collapse.
Although late fusion preserves independent channels for each modality, allowing for inference even in the absence of certain modalities, it comes at the cost of increased parameters and inference delays, which are not ideal for mobile or edge device deployment.
A more fundamental bottleneck is that these multi-modal methods have never truly addressed a key question:
Do we really understand "what are the cross-modal common liveness features"?
In other words, is the model truly learning "the authenticity of a face," or is it merely memorizing "the differences between modalities"?
Thus, a more radical question arises: If we can use multi-modal knowledge during training but only require a single modality for testing, can we build a truly "modality-agnostic" FAS system? And can such a system, while maintaining lightweight, fast, and flexible characteristics, still retain sharp judgment against spoofing attacks?
The authors of this paper propose thoughts on this and attempt to address these issues.
This time, they don't simply attach the modalities again, but rather begin by breaking them apart.
Solving the Problem
Halfway fusion can stumble when a training modality is absent, while late fusion adds branches and computation. So the authors seek a shared model that can use multimodal knowledge without requiring every sensor at test time. Here, “any modality” means one represented during training, not an arbitrary unseen sensor type:
- Absorb multi-modal knowledge during the training phase.
- Make predictions using any single modality during the testing phase.
Model Architecture
The overall architecture is shown in the diagram above. MA-ViT is built upon a single ViT backbone and contains three key modules:
- Multi-Modal Tokenization Module (on the left side of the figure)
- Transformer Encoder (indicated by the yellow and black arrows in the center)
- Modal Shared Classification Heads (the classifier on the right)
Unlike traditional multi-branch fusion designs, the key innovation of MA-ViT lies in sharing a single ViT structure across all modalities. It uses a series of standard Transformer Blocks (STB) and the newly introduced "Modality-Agnostic Transformer Block" (MATB) to stack and refine modality-agnostic representations.
The yellow arrows in the figure indicate the paths where each modality independently passes through the STB. The black-gray arrows show the subsequent processing in the MATB (MDA) module.
At the input stage of MA-ViT, whether it's an RGB or Depth image, it first passes through the Multi-Modal Tokenization Module.
As shown on the left side of the figure, the input image is divided into patches and converted into vector tokens (). These patch tokens are then combined with two special tokens:
- : The classification token, which ultimately carries the task of predicting the authenticity.
- : A modality token supervised to recognize the input type; depth is not itself a spectral band.
The sequence is added elementwise to positional embeddings () and modality-specific spectrum embeddings (), rather than combined through a separate weighted average:
These tokens will then pass into the subsequent Transformer Encoder process.
Each layer of the Transformer Encoder consists of two parts:
-
STB (Standard Transformer Block) Each modality is processed independently, as shown by the yellow and black-gray arrows, handling local features and context within a single modality.
-
MATB (Modality-Agnostic Transformer Block) Following the STB, each MATB layer is further divided into two attention modules. The implementation details are as follows:
-
MDA: Escaping the Modality
The task of Modal-Disentangle Attention (MDA) is to make the model learn to ignore patches that overly rely on modality-specific features and focus only on modality-agnostic clues strongly related to authenticity detection.
As shown on the left side of the figure, MDA first computes the modality correlation for each patch token via the dot product between the modality token (MOD) and each patch token:
Apply softmax to these similarities to obtain modality relevance weights, then let select strong positions accounting for a cumulative weight mass of . At , the threshold refers to cumulative relevance mass; the selected token count varies. Equation (4) omits that softmax in its shorthand; this explanation follows the paper’s prose and mass definition. Masked CLS-attention logits become , or a sufficiently negative implementation constant:
This CLS aggregation reads the unmasked patches. It does not physically remove patches from subsequent sequences or guarantee perfectly modality-free representations:
This is a "selective forgetting" process, where the model doesn't remember everything but only what it should.
-
CMA: Illuminating Modality Interactions
Cross-Modal Attention (CMA) creates a cross-modal sequence when multiple modalities are available during training, using the same live/spoof label for supervision. Its attention projections are shared with the single-modal path.
As shown on the right side of the figure, for example, CMA lets the query vector () from the RGB modality () perform attention operations on the key-value pairs from the Depth modality ():
The reverse direction is analogous. Both cross-modal sequences are concatenated along the batch dimension and use a residual connection. CMA shares attention and subsequent MLP weights with STB, so multimodal training influences the single-modal path; inference with one modality does not require reconstructing a missing sensor’s features.
-
Loss Design
To achieve the design where "only one modality is needed for testing," MA-ViT uses modality-shared classifiers. The CLS token predicts authenticity, and the MOD token predicts modality, with the losses of both being jointly optimized:
These equations illustrate the two-modality RGB/depth case: supervises cross-modal liveness, and modality BCE distinguishes those two modalities. Single-modal inference activates the corresponding tokenization/embeddings and single-modal path without the CMA cross-modal path.
Discussion
To validate the effectiveness of the MA-ViT framework, the authors designed two evaluation scenarios:
- Fixed Modality Test: The test modality is the same as the training modality.
- Flexible Modality Test: Any modality can independently infer the result.
In addition, cross-dataset tests and ablation experiments were conducted to gradually examine MA-ViT's ability to disassemble and translate modality information.
Fixed Modality Scenario
In the traditional setting, the model needs to perform inference under fixed modality conditions, which is also the primary comparison scenario for past methods.
For example, in the MmFA experiment, a comparison was made between the baseline (ViT without MATB) and multi-modal methods like SEF/MS-SEF:
- This ViT baseline performs below SEF on MmFA; that does not show that all Transformers cannot learn liveness without modality guidance.
- With MATB, TPR@FPR= rises from 44.33% to 82.83% and ACER falls from 6.51% to 0.80%. These outperform MS-SEF, but BPCER is 0.83% versus MS-SEF’s better 0.30%, so not every metric improves.
The comparison supports MATB’s benefit for this baseline.
Next, the experiment using the CeFA Protocol:
CeFA Protocols 1, 2, and 4 yield ACERs of 1.10%, 0.10%, and 1.64%. In Protocol 4 this beats Hulking’s 2.21% and Super’s 1.68%, but not BOBO’s 1.02%.
CeFA protocols vary ethnicity, attacks, and other conditions. The results support usefulness under those protocols.
Finally, let’s review the performance on WMCA:
The WMCA comparison uses only RGB and depth. MA-ViT achieves 1.45% seen ACER and 3.81% averaged over seven unseen attack types, versus CMFL’s 1.70% and 7.60%. Unseen flexible masks score 9.76% and glasses 14.00%, while prints and paper masks each score 0.00%. The table alone does not identify which attention mechanism produced the gains.
Cross-Dataset Generalization Ability
For MmFA→WMCA under the seen protocol, HTER is 20.63% versus Aux.(Depth)'s 24.54%; the reverse direction gives 10.41% versus 12.35%. MDA-ViT has lower HTER than CMA-ViT in both directions, 24.84% versus 27.97% and 13.30% versus 16.97%. That supports MDA's larger contribution in these tests, rather than a universal ranking of the modules.
This also echoes the original design intent of the model: Rather than pursuing modality fusion, it is more important to address modality bias first.
Flexible Modality Scenario
This is the most practical aspect of MA-ViT.
A single model trained on RGB, depth, and IR gives the following MmFA ACERs when supplied with one modality at a time:
- RGB: 11.43%, down 9.57 percentage points from MS-SEF’s 21.00%.
- Depth: 0.80%, down 2.80 percentage points from MS-SEF’s 3.60%.
- IR: 1.50%, down 17.90 percentage points from MS-SEF’s 19.40%.
These MmFA results show that one MA-ViT trained with RGB, depth, and IR can receive each modality separately at test time. It does not retrain or obtain the other modalities' inputs for that single-modal prediction.
Conclusion
MA-ViT proposes a pure Transformer face anti-spoofing system that can process each modality seen during training on its own. That flexibility matters when inputs are incomplete, although accuracy still varies by modality.
The key to this ability lies in how we redefine the role of modalities, using two core attention mechanisms within the MATB module:
- MDA: Removes modality bias and retains discriminative liveness features.
- CMA: Cross-modal complementarity, enhancing the judgment of weak local features.
Its contribution is not only an improvement in model architecture but also a shift in design philosophy:
“Rather than remembering modalities, it’s better to perceive the essence.”
A model should not just memorize what modality you give it. It should learn to see what matters in each input it supports.