CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing
- Publication
- WACV 2021Conference
- First submitted to arXiv
- arXiv:2003.05136
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{faceantispoofingcefa2020,
title = {{CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing}},
author = {Ajian Li and Zichang Tan and Xuan Li and Jun Wan and Sergio Escalera and Guodong Guo and Stan Z. Li},
year = {2020},
eprint = {2003.05136},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2003.05136}
}
The CASIA-SURF in the paper title refers to a face anti-spoofing dataset created by the Chinese Academy of Sciences, Institute of Automation (CASIA), called:
CASIA-SURF: A Large-Scale Multi-Modal Benchmark for Face Anti-Spoofing
This dataset was released in 2019 and became one of the key datasets created for multi-modal face anti-spoofing at the time.
This paper presents an extended version of the dataset aimed at addressing ethnic bias, named CASIA-SURF CeFA.
Defining the Problem
In the field of Face Anti-Spoofing (FAS), everyone is striving for models that are "more accurate, faster, and more generalized."
But an old issue rarely received the same attention: Ethnic Bias.
Had people forgotten about it, or were they avoiding it?
If you have worked on facial recognition, you are definitely familiar with this issue.
Some face recognition datasets have unbalanced demographic representation, and error rates can differ across groups. Training-data coverage matters, but it is not the only factor; such differences cannot simply be attributed to ethnicity itself.
Improving representation and evaluating errors separately by group are important steps. Bias reduction still needs evaluation by group.
This issue is also present in FAS.
The authors evaluated two representative models of the time on CeFA and found substantial differences across groups:
For the two models, ACER on the Central Asian group exceeded that on the East Asian group by 8.2 and 10.7 percentage points, respectively.
ACER combines attacks accepted as genuine and genuine users rejected as attacks. It does not tell us which error is responsible; APCER and BPCER must also be examined.
Now imagine that genuine users in one group keep seeing “Fake! Fake! Fake!” Would they be tempted to throw the system—and its deployment engineer—into the river?
Solving the Problem
CeFA Dataset
The CeFA dataset is compared with other datasets in the table below:
CeFA uses the Intel RealSense camera to capture three-modal video data, including RGB (color), Depth (depth), and IR (infrared). Each video frame has a resolution of 1280 × 720, captured at 30 fps.
During the recording, participants were asked to naturally rotate their heads, with a maximum deviation angle of about 30 degrees, simulating natural facial dynamics in real-world scenarios.
The preprocessing steps for the data are as follows:
-
Face detection uses the 3DFFA model (replacing PRNet).
-
The aligned facial regions are extracted and the corresponding three-modal samples are retained.
-
Multimodal facial samples are shown in the image below:
The CeFA dataset consists of the 2D Attack Subset and 3D Attack Subset, covering three groups (African, East Asian, and Central Asian), with varied attacks and lighting. These are broad dataset categories, not homogeneous populations or a complete representation of global demographic diversity.
-
2D Attack Subset (Print & Replay)
- Number of participants in each group:
- 500 participants per group, for a total of 1500 participants
- Samples per participant:
- 1 real sample
- 2 print attack samples presented on printed cloth (one indoors, one outdoors)
- 1 video replay attack sample
- Modal data:
- RGB, Depth, and IR modalities are captured synchronously for each sample
- Total number of samples:
- 1500 participants × 4 video clips = 6000 clips × 3 modalities = 18,000 video clips
The paper also provides demographic statistics on age and gender for participants in the 2D subset, which will be helpful for future fairness analysis.
- Number of participants in each group:
-
3D Attack Subset (3D Masks and Silicone Masks)
This subset simulates more realistic advanced attack scenarios, including two types of attacks:
-
3D Printed Mask Attacks
- Number of participants: 99 people
- Samples per participant:
- 3 types of attack styles:
- Plain mask
- Wig + glasses
- Wig without glasses
- 6 lighting conditions:
- Outdoor sunlight, outdoor shade
- Indoor side light, front light, back light, regular light
- A total of 18 video clips × 3 modalities
- 3 types of attack styles:
-
Total number of samples: 99 × 18 × 3 = 5346 video clips
-
Silica-Gel Mask Attacks
- Number of participants: 8 people
- Samples per participant:
- 2 types of attack styles:
- Wig + glasses
- Wig without glasses
- 4 lighting conditions:
- Indoor side light, front light, back light, regular light
- A total of 8 video clips × 3 modalities
- 2 types of attack styles:
-
Total number of samples: 8 × 8 × 3 = 192 video clips (the paper's prose says 196, but Table 1 and the stated total of 23,538 use 192)
These counts treat modalities as separate video streams: 18,000 + 5,346 + 192 = 23,538. Synchronously captured RGB, Depth, and IR streams are not three independent attack events.
Although the 3D subset is smaller, its varied masks and capture conditions make it a useful test of generalization.
-
Evaluation Protocols
In order to systematically evaluate the generalization ability of models in different scenarios, the CeFA dataset designs four evaluation protocols, covering challenges such as cross-ethnicity, cross-attack types, cross-modality, and their combinations.
The table below shows the specific configurations for each protocol:
The protocols use the 2D subset for training and validation and include the 3D attacks in their test sets, as specified in the paper's Table 2 caption.
-
Protocol 1: Cross-ethnicity
This protocol addresses the common issue in existing PAD datasets, which lack ethnicity annotations and cross-group testing setups. It is designed to evaluate the model’s generalization ability when faced with previously unseen ethnic groups.
The approach involves selecting one ethnic group for training and validation, and using the remaining two groups' data as the test set. This results in three different experimental configurations.
This setup simulates a real-world scenario where a model is developed for a specific ethnic group and then deployed in environments with other ethnic groups.
-
Protocol 2: Cross-PAI
This protocol tackles the diversity and unpredictability of presentation attack instruments (PAI) in attack forms, evaluating the model’s robustness when faced with unknown attack styles.
During the training and validation stages, only certain attack types are used, while the test stage employs previously unseen attack methods to observe whether the model can identify different types of spoofing behavior.
-
Protocol 3: Cross-modality
This protocol is designed to explore the model's generalization ability across different modalities. During the training stage, only one modality (RGB, Depth, or IR) is used, while the test stage evaluates the remaining two modalities.
This setup is designed to test the model’s performance when sensor equipment is limited, or modality information is incomplete, simulating real-world scenarios where cross-modal feature alignment is lacking.
-
Protocol 4: Cross-ethnicity & Cross-PAI
This is the most challenging protocol, combining the conditions of Protocol 1 and Protocol 2. The test set includes both unknown ethnic groups and unknown attack types.
This setup combines two distribution shifts. It tests unseen groups and attack types under specified conditions, rather than covering every possible deployment environment.
Model Architecture
For the CeFA dataset, the model proposed in this paper consists of two major modules:
- SD-Net: A single-modality static-dynamic feature extraction network that focuses on capturing spatial and temporal information within a single modality.
- PSMM-Net: A partially shared multimodal network designed for feature fusion and interactive learning across different modalities.
SD-Net
SD-Net processes single-modality inputs (RGB, Depth, or IR), and its core design aims to simultaneously capture both static image and dynamic sequence information. The overall architecture consists of three feature branches:
- Static Branch (red arrow): Handles single static images to extract spatial features.
- Dynamic Branch (blue arrow): Summarizes K-frame sequences using Rank Pooling, converting them into a dynamic feature map.
- Static-Dynamic Branch (green arrow): Fuses the above two features, enhancing the interaction between spatial and temporal information.
The branches are based on ResNet-18. The Static and Dynamic branches contain five blocks—conv and res1–res4—followed by global average pooling (GAP), not five residual modules. The Static-Dynamic branch omits conv and res1 and starts from the sum of the two res1 outputs, followed by res2–res4.
Each module is denoted as , where:
- : Modality type;
- : Feature layer levels.
Each branch produces feature maps at the corresponding levels, denoted as:
- (Static)
- (Dynamic)
- (Fusion)
To encourage the three branches to learn complementary information, the authors designed independent loss functions for each branch (using BCE) and also introduced a loss for the combined features:
PSMM-Net
Next, let's look at the PSMM-Net portion.
The goal of PSMM-Net is to capture common semantics and complementary information across modalities. It consists of two parts:
- Modality-specific Networks: Three SD-Nets, each corresponding to RGB, Depth, and IR modalities, independently extracting static and dynamic features.
- Shared Branch: Based on a ResNet-18 architecture (removing the conv and res1 layers), used to model shared semantics across modalities.
To achieve effective modality interaction, PSMM-Net employs a bidirectional fusion strategy:
-
Forward Feeding (to shared branch)
At , the static and dynamic features from the three modalities are summed and fed into the next shared-branch block:
Where is the output of the shared branch at layer (, starting from zero).
-
Backward Feeding (from shared branch back)
At , is added to each modality’s features before the next SD-Net block:
This step applies to the static and dynamic branches; it does not feed directly into the fusion branch.
SD-Net processes static and dynamic information in separate branches, while PSMM-Net exchanges features across modalities. The protocols and ablations assess disentanglement and generalization.
The final loss function is:
Where is the total loss after summing the features from all modalities and the shared branch.
Discussion
Model Bias Analysis Under Ethnic Diversity
The authors conduct a performance bias analysis on three different ethnic groups (East Asian, Central Asian, African) in the CeFA dataset, observing the performance differences of current mainstream models when facing ethnic variations.
The experiment uses two state-of-the-art (SOTA) models trained on two public datasets, representing multimodal and single-modality anti-spoofing methods:
- MS-SEF: A multimodal method trained on CASIA-SURF (mainly East Asian samples)
- FAS-BAS: A single-modality RGB method trained on OULU-NPU (also primarily East Asian)
After training, both models are tested on the three ethnic groups in the CeFA dataset, using ACER as the primary evaluation metric.
Both models have their highest ACER on the Central Asian group. MS-SEF obtains 13.9%, 19.6%, and 11.4% on African, Central Asian, and East Asian samples; FAS-BAS obtains 14.2%, 26.1%, and 15.4%, respectively.
The Central Asian–East Asian gaps are therefore 8.2 and 10.7 percentage points. The best-performing group differs: East Asian for MS-SEF, African for FAS-BAS. Training-set composition may contribute, but the table alone cannot isolate ethnicity as the cause.
These results show why aggregate scores should be supplemented with group-level evaluation such as Protocol 1. A fairness analysis should also control capture conditions, attacks, and data sources, and separately report genuine-user rejections and accepted attacks.
Baseline Model Evaluation Results
The authors conduct comprehensive experiments on the SD-Net and PSMM-Net models under the four evaluation protocols designed in CeFA:
-
Protocol 1: Cross-ethnicity
- Sub-protocols 1-1, 1-2, and 1-3 have ACER values of 0.6%, 4.4%, 1.5%, respectively.
- The three unseen-group combinations yield different results, supporting separate reporting for each cross-group setting. This experiment does not compare training on mixed ethnic groups, so it cannot quantify the benefit of doing so.
-
Protocol 2: Cross-PAI (Generalization Across Different Attack Tools)
-
Sub-protocol 2-1 (trained on print, tested on video-replay + 3D) has an ACER of 0.4%.
-
Sub-protocol 2-2 (trained on video-replay, tested on print + 3D) has an ACER of 7.5%.
-
The two training and test attack combinations yield very different results. The table alone cannot attribute this gap specifically to screen displays or print materials.
-
-
Protocol 3: Cross-modality (Generalization Across Modalities)
-
The best result is from sub-protocol 3-1, with an ACER of 4.9%.
-
The other two sub-protocols yield ACERs of 13.6% and 11.7%. Transfer between modalities varies with the combination; this table alone cannot isolate semantic alignment as the cause or demonstrate the benefit of multimodal fusion.
-
-
Protocol 4: Cross-ethnicity & Cross-PAI (Dual Generalization Test)
- Sub-protocols 4-1, 4-2, and 4-3 have ACER values of 24.5%, 43.2%, 27.7%, respectively.
- Among the four protocols, Protocol 4 has the highest mean ACER: this baseline struggles most when the group and attack type both change. The result is a CeFA benchmark measurement, not a direct estimate of errors after deployment.
Conclusion
The authors position CeFA as the largest multimodal face presentation attack dataset with ethnicity labels at the time. Its RGB, Depth, and IR data explicitly bring cross-ethnicity generalization into the evaluation. This is the paper’s historical positioning.
PSMM-Net’s partial sharing and bidirectional fusion, together with SD-Net’s separate static and dynamic branches, were a fairly systematic approach to multimodal learning at the time. Later architectures may make different tradeoffs, but the question of what to share across modalities remains.
Looking back, the key contribution of CeFA is clearly defining the research challenges of "cross-modality, cross-ethnicity, and cross-attack types," providing a reasonable starting point. For us today, this is not just a dataset but a continuously extending problem space.