La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection
- Publication
- IJCB 2024Conference
- First submitted to arXiv
- arXiv:2408.12793
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofinglasoftmoe2024,
title = {{La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection}},
author = {Hang Zou and Chenxi Du and Hui Zhang and Yuan Zhang and Ajian Liu and Jun Wan and Zhen Lei},
year = {2024},
eprint = {2408.12793},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2408.12793}
}
Physical attacks (PAs) range from color photos spit out by printers to deep eye sockets beneath lifelike masks.
Digital attacks (DAs) range from phantoms generated by algorithms to direct manipulation of the image record itself.
The boundary of facial recognition systems has never been so blurred.
Problem Definition
In past research, these were treated as separate tasks:
-
PAD (Physical Attack Detection):
Focused on multimodal sensing and physiological feature extraction, emphasizing measurable aspects of the physical world;
-
DAD (Digital Attack Detection):
Concentrated on detailed feature-level analysis, frequency domain anomalies, and visual deception patterns.
Although each has made progress, this separation has led to repeated model deployments and difficulty in cross-task transfer.
In practical scenarios, no system encounters only one attack type, making Unified Attack Detection (UAD) an urgent and natural research direction.
Unifying the inputs still leaves the problem of feature fusion.
Looking at the feature distributions shown by UniAttackData, the two attack types distribute very differently in embedding space, especially under ID-consistency settings. Even attack samples from the same face demonstrate isolated clustering tendencies.
This "sparse and split feature space" is a blind spot for existing methods.
While some researchers have tried multi-branch designs (e.g., CA-MoEiT), multi-task learning, or introducing physiological information to handle the two attack types, these methods often introduce new challenges:
- Increased architectural complexity, leading to higher computational resource consumption and deployment latency;
- Lack of ID-consistent datasets causing models to learn background and non-semantic cues;
- In large-scale general models (such as CLIP), feature sparsity among dimensions further weakens the stability of classification logic.
In such contexts, even vision-language models like CLIP struggle to recognize a unified syntax for "fake faces" within a sparse feature space.
People start to realize: this might not be merely a lack of data, but rather that semantic feature distributions are overly fragmented. The real problem may not lie in model capacity but in the absence of clear methods for the model to utilize its own capabilities.
Perhaps it is time for the model to learn self-scheduling and self-collaboration.
This is exactly where the design logic of Mixture-of-Experts (MoE) originates.
Solution
To address the above issues, the authors propose an architecture that combines the Mixture-of-Experts (MoE) design philosophy with a linear attention mechanism, named La-SoftMoE CLIP.
This method modifies the original CLIP architecture, specifically enhancing its classification capability under sparse semantic distributions.
SoftMoE Module Design
To strengthen CLIP’s recognition ability in sparse semantic space, the authors introduce the Soft Mixture-of-Experts (SoftMoE) module, embedded within the residual blocks of each ViT layer as shown above.
Compared to traditional MoE architectures, SoftMoE has the following advantages:
- Soft Dispatch: Uses softmax weights to map all input tokens to slots, reducing the token-dropping risk of top- hard assignments; this paper does not separately measure expert load balance.
- Differentiable Design: The entire operation is fully differentiable, enabling stable joint training with the backbone network.
- Scalability potential: No sorting or top- expert selection is required. The authors see room to scale the number of experts, but do not test thousands of them here.
The specific computational process is as follows:
-
Dispatch Weights
In Soft-MoE, the traditional top- dispatch strategy is abandoned and replaced by:
- Soft dispatch mechanism
The core idea is to introduce a set of intermediate representations called slots, which serve to aggregate the semantic weighted average of input tokens. Each slot does not correspond to a single token but is generated as a weighted sum of tokens, effectively representing a semantic subspace aggregation vector.
Each expert handles several slots, allowing the model to perform differentiated expert computations based on data characteristics.
The dispatch process is as follows: Given the input token matrix , a learnable parameter set projects into dispatch logits, and a column-wise softmax is applied over each slot ( slots):
- is the dispatch weight matrix, representing the contribution of token to slot .
- is the input representation matrix for all slots, where the -th row represents the weighted semantic vector aggregated by slot .
This design integrates information from all tokens into each slot, weighted by different attention intensities. The model thus transforms discrete token-level features into higher-level, shareable slot-level representations.
This soft dispatch reduces the token-dropping risk of traditional hard routing, laying a differentiable feature foundation for subsequent expert processing and combination weights.
-
Expert Processing and Combine Weights
is grouped by slots and sent into Experts (each expert processes slots) to produce slot outputs .
Then, Combine Weights are computed. This time a row-wise softmax is applied per token to decide “how to gather information back from each slot”:
- is the output token list of Soft-MoE. In this paper, the module sits after ViT attention and in parallel with the MLP.
La-SoftMoE
Although Soft-MoE features differentiable dispatch and recombination mechanisms, its combination phase still has a potential limitation:
In the original design, each output token’s weighting over all slots comes from a row-wise softmax of the same logits used for dispatch. Different slots still receive different weights. The authors argue that this fixed form of weighting may not account sufficiently for differences among expert outputs.
To solve this, the authors propose an improved version: La-SoftMoE, implementing two key modifications:
-
Replace row-wise softmax with a Linear Attention mechanism
The authors instead use a feature map to compute attention so that the combine weights can take slot representations into account. This kind of linear attention aims to reduce the cost of attention as sequence length grows; the paper does not report deployment measurements across sequence lengths.
-
Add Instance Normalization in weight calculation
La-SoftMoE applies Instance Normalization after the Linear Attention output and before the combine weights are used. The paper does not ablate this step alone, so the experiments cannot isolate its contribution to stability or generalization.
Implementation Details
- Embedding position: La-SoftMoE modules are embedded in the 12 residual blocks of the ViT, arranged in parallel with the original Feed-Forward MLP submodules.
- Expert configuration: Each layer has experts, each handling slots, totaling slots; larger configurations require separate memory and latency measurements.
- Overhead evaluation: The paper compares detection metrics, but does not report inference latency relative to the original ViT or an F1-score gain.
- Implementation notes: The pseudocode uses
einsumfor the token/slot matrix operations and applies Instance Normalization after Linear Attention.
Through the above architectural adjustments and module integration, La-SoftMoE CLIP lets different experts participate in modeling PA and DA features. The paper does not quantify the additional parameter count or total inference cost.
On the two tested benchmarks, the method improves ACER over the original CLIP. Generalization to an unseen target domain would require a separate leave-out-domain protocol.
Discussion
The experiments are primarily based on UniAttackData, the first benchmark for Unified Attack Detection with ID consistency, which includes:
- 1800 subjects covering 2 types of physical attacks (PAs) and 12 types of digital attacks (DAs)
- The first evaluation protocol uses complete live/fake paired samples for training and testing
Supplementary experiments use JFSFDB, which integrates 9 FAS sub-datasets, including traditional PAD and deepfake DAD examples.
The main protocol employed is Joint Training across the constituent datasets; it is not a test on a held-out target dataset.
The model backbone is ViT-B/16 with embedded La-SoftMoE modules (4 experts per layer, 49 slots), trained on an A100 GPU using Adam optimizer with a learning rate of .
Experimental Results Analysis (UniAttackData)
The table above compares La-SoftMoE CLIP with existing methods, showing state-of-the-art performance on key metrics:
- ACER: 0.32% (previous best 0.52%)
- Accuracy: 99.54% (previous best 99.45%)
- AUC and EER slightly lower than the top model but still very high (99.72% / 0.56%)
This indicates a lower detection error rate on this dataset. Whether the feature space itself is more compact calls for the visualization below.
Mixed-Dataset Test (JFSFDB)
To examine performance on another benchmark mixing physical and digital attacks, the authors also test JFSFDB under its joint-training protocol. This does not hold out a dataset as an unseen target domain. Its numbers cannot be compared directly with UniAttackData under identical conditions; La-SoftMoE CLIP achieves:
- ACER: 4.21%
- Accuracy: 95.85%
- AUC: 99.11%
- EER: 4.19%
The authors also visualized feature distributions of both datasets with t-SNE (see below):
The UniAttackData points appear more concentrated and connected along curves, whereas the JFSFDB points are more dispersed. The authors suggest ID consistency may be one reason.
Prompt Robustness Test
Given that CLIP-like models are sensitive to language prompts, the authors designed 8 semantically equivalent but structurally different prompts (T-1 to T-8, see table below) to evaluate La-SoftMoE CLIP’s stability and semantic adaptability.
Results show:
- T-8 ("This is an image of a <fake/live> face") has the lowest ACER and ties T-3 for the highest Accuracy
- T-3 ("A photo of a <fake/live> face") yields highest AUC and lowest EER
The figure below compares performance differences across prompts:
Across the eight prompts, ACER ranges from 0.32% to 0.63%, a visible difference. This test shows the model works with these different wordings.
Ablation Study
Table 6 sequentially compares:
- Vanilla CLIP
- CLIP + SoftMoE
- CLIP + La-SoftMoE
In this test, SoftMoE and then La-SoftMoE successively lower ACER and EER and raise Accuracy. Both have lower AUC than vanilla CLIP’s 99.76%, however. The table changes both Linear Attention and Instance Normalization, so the gain cannot be attributed to the former alone.
Finally, feature distributions of four models are shown:
- ResNet-50 (top left)
- CLIP (top right)
- CLIP + SoftMoE (bottom left)
- CLIP + La-SoftMoE (bottom right)
In this t-SNE view, some CLIP live and fake points mingle. With SoftMoE, the selected samples appear more separated; in the La-SoftMoE view, the two groups also appear broadly apart.
Together, the reported results show better detection metrics on two mixed-attack benchmarks. The feature structure and computational cost still need more direct measurement.
Conclusion
In this paper, the authors introduce Soft Mixture-of-Experts (SoftMoE) into the CLIP image encoder and further refine its weighting strategy using linear attention and normalization mechanisms. This lets the model handle heterogeneous features of physical and digital attacks; the computational overhead is not quantified here.
The key insight is: when semantic distributions are intrinsically fragmented, rather than forcibly unifying them, it is better to let the model learn to allocate experts and share responsibilities, reorganizing the feature space structurally.
Future directions include exploring semantic-driven routing mechanisms, prompt-conditioned generation, and token-slot alignment methods with cross-modal generalization capabilities to further expand structured designs’ potential in multimodal recognition tasks.