Multi-Modal Face Authentication using Deep Visual and Acoustic Features
- Publication
- ICC 2019Conference
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@inproceedings{faceantispoofingvafas2019,
title = {{Multi-Modal Face Authentication using Deep Visual and Acoustic Features}},
author = {Bing Zhou and Zongxing Xie and Fan Ye},
year = {2019},
booktitle = {ICC},
url = {https://doi.org/10.1109/icc.2019.8761776},
doi = {10.1109/icc.2019.8761776}
}
In this paper, the authors did not provide a concise name for their method, so we’ve extracted features from both Visual and Acoustic elements and named it VA-FAS.
Defining the Problem
Smartphones have already incorporated "face recognition" into the daily startup sequence.
To prevent photos from fooling the camera, mainstream solutions capture temporal cues like blinking, lip movements, and subtle head motions in video streams, or detect spatial anomalies in the image to determine whether the user is real.
It seems like no issues, but in reality, if an attacker uses high-resolution video, some temporal and spatial cues can be reproduced, defeating methods that rely on them.
When simple RGB cues cannot uphold the security boundary, the industry moves towards a second approach: adding more sensors next to the camera.
For instance, iPhone’s Face ID uses near-infrared structured light to reconstruct 3D geometry, raising the attack threshold; some flagship Android phones also add ToF or binocular modules.
However, this "hardware reinforcement" strategy may face three challenges:
- Cost: Dedicated modules increase the BOM, making it difficult to scale to mid-range or low-end devices.
- Space: The front camera area is precious, adding another sensor means sacrificing part of the screen.
- Ecosystem Lock-in: Sensor specifications are deeply coupled with algorithms, making cross-platform replication challenging.
The paper focuses on phones without a dedicated depth sensor, using their existing front camera, speaker, and microphone.
Thus, we need a third path: Injecting depth into face recognition without adding extra hardware.
As a result, the authors turned their attention to sound waves:
The reflected signals a phone already has the hardware to measure.
Can sound fill in a layer of depth for 2D images? Can the combination of images and sound truly seal the gap for video attacks?
Solving the Problem
The authors proposed an acoustic-visual fusion face authentication system that enhances liveness verification strength on mobile phones without adding new hardware.
The overall system consists of three modules:
- Acoustic feature extraction
- Image feature extraction
- Multi-modal feature fusion and classification
Both acoustic and image features are extracted using CNNs and then concatenated into a joint vector, which is classified using a One-Class SVM.
The overall architecture is quite simple. Let's dive into the details of each module.
Acoustic Feature Extraction
The authors chose to use the built-in "earpiece speaker + top microphone" as the acoustic communication pair. This setup is widely available on various mobile devices and, when placed alongside the front camera, reduces misalignment between the acoustic and visual sensors.
The sound waves transmitted are 16–22 kHz linear chirp, which the authors describe as almost inaudible to most users. The 16 kHz end may still be audible to some people, so the entire band should not be called ultrasound. It belongs to the FMCW (Frequency-Modulated Continuous Wave) type, which estimates the reflection path length through frequency differences.
To improve signal quality, the system implements the following designs:
- Using short pulses to avoid self-interference from the speaker;
- Applying a Hann window (called Hanning in the paper) to smooth the signal edges and enhance time-frequency stability.
The acoustic signal processing flow is as follows:
The recorded signal first passes through a Butterworth band-pass filter (16–22 kHz) to remove background noise.
The processed waveform can be observed as:
- Direct path: The direct signal from the speaker to the microphone, usually with the highest amplitude.
- Facial echo: The main signal reflected from the forehead, nose bridge, cheeks, etc.
- Noise and interference reflections: Reflections from the desk, collar, etc., which need to be excluded.
The segmentation strategy is as follows:
- Peak detection is used to locate the direct path;
- Cross-correlation is then used along with a template signal from the registration phase to lock onto the main echo region.
If multiple similar peaks cause instability in the positioning, the facial distance information estimated by the front camera is used to narrow the search range. For instance, if the distance between the eyes widens in the image, it indicates that the face is closer, and the expected echo position can be adjusted.
The final extracted segments are extended by 10 samples on both sides to retain a depth range of about 7 cm.
The echo signal undergoes an additional FMCW distance transformation, as shown in the diagram:
The FMCW technology converts the time delay into frequency shift, and under a colocated transmitter/receiver approximation, ignoring motion-induced frequency shifts, the one-way distance can be written as follows (an explanatory formulation of the paper's concept):
Where:
- = distance;
- = speed of sound;
- = frequency difference between the received and transmitted signals;
- = chirp duration;
- = bandwidth (6 kHz).
The frequency difference obtained through "signal mixing" is then converted into a time-frequency plot (spectrogram).
This time-frequency plot is fed into a pre-trained CNN (based on the EchoPrint architecture) for inference, with the final classification layer removed, and output as a 128-dimensional vector representing echoes affected by facial geometry and material, rather than an explicitly reconstructed 3D face.
Image Feature Extraction
After the acoustic feature extraction, the next step is image feature extraction, which involves four steps:
- Face Detection: Using HOG and a pre-trained model to detect the face box and facial landmarks.
- Affine Transformation: Aligning the facial landmarks to a standard template.
- Cropping and Resizing: Obtaining a fixed-length input image.
- Feature Extraction: Feeding the cropped, aligned face into the image CNN.
For feature extraction, a simplified version of OpenFace (based on FaceNet’s nn4) is used, along with classification loss + triplet loss training. The final output is a 128-dimensional image feature vector.
Feature Concatenation
The two feature vectors are simply concatenated to form the final 256-dimensional joint feature:
The classifier used is One-Class SVM, which builds a high-dimensional boundary with only the registered samples, treating other users as anomalous data. The advantage of this approach is that it doesn’t require the collection of attack samples, and the personalized classifier is relatively lightweight. The prototype still offloads visual feature extraction to a server; acoustic processing and final inference run on the phone, so the full pipeline is not offline.
The final system is shown below; the user aligns their face within the green valid area, and the prototype collects acoustic and visual data before making an authentication decision. The paper does not report a guaranteed 300 ms end-to-end latency; image transfer and server delay depend on the network.
Discussion
Data and Experimental Design
-
Subjects: 10 volunteers (not included in the acoustic model training set).
-
Acoustic Samples: Approximately 2 minutes of recording per person, with a total of 13,806 samples.
-
Image Samples: 20 face images per person, with interpolation between neighboring images to match the acoustic sample count.
-
Attack Samples: 5 types of fake faces, including:
- Paper photos, screen displays, cardboard surfaces, wall posters, and statues.
Data was collected in uncontrolled environments, covering various light sources and background noise to test the system’s practicality.
The model performance is measured using four indicators:
Where:
- High Precision → A high proportion of accepted samples belong to the legitimate user, treated here as the positive class. Precision is not FAR and depends on class proportions.
- High Recall → Indicates fewer false rejections of legitimate users.
- F1 and BAC summarize performance. F1 still depends on class proportions; BAC gives equal weight to positive- and negative-class recall.
The data is randomly split 80:20 for training and testing. Because images are interpolated and acoustic samples can come from adjacent time segments, this is not an independent cross-session or cross-device evaluation. The model uses One-Class SVM, with each user training on their own samples, with other people's data as negatives in the identity experiment and separate photo/screen presentation-attack tests. Each modality (acoustic, image, combined) follows the same procedure to ensure fairness in comparison.
Bimodal Verification Performance
Based on the bimodal architecture, the experimental results are shown in the table above, with the left value in each cell showing the mean and the right value showing the median.
In this genuine-user verification test, joint acoustic and visual features achieve a mean F-score of 94.07% and BAC of 94.42%, but do not lead on every metric. Mean precision is 99.96%, slightly below vision alone at 100%; mean recall is 88.84%, below acoustics alone at 89.83%. The authors also test the prototype against photos and screen-displayed faces and observe no successful attacks in their tests, while acknowledging that larger attack experiments are still needed.
The joint-feature model also distinguishes one pair of twin sisters. This is a preliminary result for one pair, not enough to establish general twin-recognition performance or show which modality was decisive.
Impact of Background Noise
The practicality of the acoustic module depends on whether it can function stably in real-world environments. To test this, the authors repeatedly performed data collection and recognition tests in noisy laboratory and classroom environments with human speech.
The authors observed no obvious performance degradation under these two background-noise conditions. They do not provide separate numerical results for each setting.
Conclusion
This paper is one of the early explorations of "sound waves as a liveness cue" in practical implementations, and it provides a feasible solution given the hardware constraints of the time.
It did not lean too heavily on new hardware, nor did it assume a more ideal set of data and devices. The prototype still depends on alignment, pretrained models, and server-side visual feature extraction; its ten-person experiment supports feasibility rather than a product-level security guarantee.
Instead, it presents an engineering reality, raising a critical question given resource limitations:
If a smartphone only has a microphone, speaker, and front camera, how much defensive space do we have left?
The answer this research tests is: for this prototype, the face in the sound waves was enough to help.