FRVT: Face Recognition Verification Accuracy on Distinguishing Twins (NIST.IR.8439)
- Publication
- NIST IR 8439 2022Technical report
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@techreport{facerecognitionfrvtdistinguishingtwins2022,
title = {{FRVT: Face Recognition Verification Accuracy on Distinguishing Twins (NIST.IR.8439)}},
author = {Kayee Hanaoka and Mei Ngan and Patrick Grother and Austin Hom},
year = {2022},
institution = {NIST IR 8439},
url = {https://doi.org/10.6028/NIST.IR.8439},
doi = {10.6028/NIST.IR.8439}
}
This is not a paper but a technical report.
NIST studied the distinction between twins in its Face Recognition Vendor Test (FRVT).
This is interesting: twins look alike, so face recognition systems might have some trouble...
Wait, that understates it. For these systems, "distinguishing twins" is a suffocatingly difficult task.
Introduction
Facial recognition technology is increasingly used in both public and private sectors for identity verification, transaction authorization, and access control.
Over the past decade, accuracy in the FRVT evaluations has significantly improved, supporting these applications.
Since the COVID-19 pandemic, some algorithms have even been able to recognize individuals wearing masks.
However, despite these advancements, the identification of twins remains problematic.
The report covers algorithms submitted through mid-February 2022; its findings do not describe every present-day system.
Datasets
Twins Day Dataset 2010-2018
This dataset, from West Virginia University’s Biometrics and Identification Innovation Center, includes images from Twins Days from 2010 to 2018.
These collections were conducted under IRB protocols with informed consent from each participant.
The image sizes vary by year:
- 2010: 2848x4288
- 2011: 3744x5616
- 2012 to 2018: 3300x4400 and 2400x3200.
The images are high-quality frontal portraits, meeting NIST SAP50 specifications (head and shoulder composition) and SAP51 specifications (head composition). The two examples below are report Figures 2 and 3; their captions identify a Notre Dame twins database, so they should not be presented as verified samples from the WVU evaluation set.
-
Sample Image: Different Individuals
-
Sample Image: Same Individual
Immigration-related Images
This dataset includes real-time captured images, primarily from webcam images and some visa application images.
Webcam images are taken by operators under time constraints, leading to variations in roll, pitch, and yaw angles, with faces sometimes underexposed against bright backgrounds, perspective distortion from close capture, and partial cropping. The average inter-pupil distance is 38 pixels, not conforming to ISO/IEC 19794-5 full-frontal image types.
Some subjects also have visa application images captured in interview environments using dedicated equipment and lighting. These images conform to ISO/IEC 19794-5 standards with a capture size of 300x300 and nearly frontal poses.
-
Example Image
Dataset Limitations
Small Population
- The report uses 5,938 images of 698 twin pairs across both datasets combined; that is not the size of the Twins Days set alone. Together the sets are small relative to other NIST FRVT 1:1 datasets.
- Due to inconsistent or missing metadata, many images and twins were excluded, including some incorrect matches (e.g., A and B are twins, but B and C are also recorded as twins).
- Most of the immigration-related dataset images were retained, including 152 pairs of twins and 2,478 images.
Variations in Identification Codes
- Twins Days usually assigns the same identification code to the same participants each year, but sometimes a new code is issued.
- This results in some high-threshold non-twin comparisons that are actually correct matches being incorrectly marked as errors.
Incorrect/Missing Metadata
- Most or all metadata is missing for the 2011, 2014, 2015 and 2016 Twins Days collections.
- Incorrect metadata, such as twin type or birthdate mismatches, led to the exclusion of many images.
- The immigration-related image dataset lacks information to distinguish fraternal from identical twins.
Imbalanced Data
- Twins Days images have an uneven distribution of twin types: 2.8% fraternal opposite-sex, 6.7% fraternal same-sex, and 90.5% identical twins.
- Age group distribution is also uneven, with most images belonging to the 20-39 age range, and very few from the 40-59 and 60+ age groups.
Racial Imbalance
- The Twins Days dataset includes racial identity: 85% White, 10% African American, and 5% others.
- The groups have highly unequal sample counts, limiting reliable comparisons between them.
Algorithm Performance Report
With FRVT being open to global participation, many algorithms are involved.
This report documents the results of 478 algorithms submitted to the FRVT 1:1 test from 2019 to mid-February 2022.
Evaluation Metrics
-
False Match Rate (FMR)
The False Match Rate (FMR) is the proportion of different-identity comparisons incorrectly accepted as the same person.
This is a crucial performance metric in facial recognition systems, especially in high-security applications like banking or airport security.
The report selects a threshold that gives FMR = 0.0001 on non-twin comparisons, then measures twin FMR at that threshold. Twins Days uses a general mugshot control, while immigration-related comparisons use a visa-border control; 0.0001 is not the measured twin FMR.
-
False Non-Match Rate (FNMR)
The False Non-Match Rate (FNMR) is the proportion of same-identity comparisons incorrectly rejected as different people.
This is also an important performance metric, particularly in applications concerning user convenience and experience.
-
FNMR @ FMR = 0.0001
When you see "FNMR @ FMR = 0.0001," it means measuring the same-identity false non-match rate at the threshold calibrated to FMR = 0.0001 on the relevant non-twin control comparisons.
This checks ordinary verification errors at a particular operating point; twin FMR must be measured separately.
Identical Twins
-
At the threshold calibrated using unrelated people, the FMR for identical twins in Twins Days usually exceeds 0.99, indicating that most comparisons between the two members of a twin pair are incorrectly accepted as matches.
-
FMRs below 0.99 generally result from two situations:
- System Errors: Algorithms with high Failure to Enroll (FTE) rates failing to extract features from many or all images.
- Poor Algorithms: Systems with high False Non-Match Rates (FNMR), producing low scores, meaning the algorithm sees everything as different people.
Algorithms Capable of Identifying Identical Twins
Some algorithms show high accuracy in recognizing identical twins, not always mistakenly identifying identical twins as the same person.
These algorithms include:
- Algorithms: aigen-001, aigen-002, beyneai-000, glory-004, mobai-000, and iqface-001.
- Performance Metrics:
- Mugshot-control FNMR (False Non-Match Rate): ≤ 0.02
- Mugshot-control FTE (Failure to Enroll Rate): ≤ 0.02
- Identical-twin FMR (False Match Rate): ≤ 0.7
Specifically, aigen-002 is noteworthy, with an FMR of 0.475 for identical twins, meaning this algorithm correctly distinguishes twins about half the time. While this FMR is much higher than the nominal FMR used to calibrate the threshold on unrelated people (0.0001), it is significantly lower than the 0.98 and 0.99 FMRs of most other algorithms, indicating better performance in distinguishing identical twins.
Comprehensive Analysis
Score Distributions
The figure above shows score distributions for idemia-008 and paravision-010 across different pair types. Their score scales differ, so absolute x-axis values should not be compared directly between algorithms.
The highest scores come from "mated twins," indicating comparisons between photos of the same individual taken on the same day.
The next highest scores are from "mated mugshot," images of the same person in the general mugshot control group.
Comparisons between identical twins (labeled "non-mated identical twins") also receive similarly high scores.
Scores for same-sex fraternal twins (labeled "non-mated fraternal same-sex twins") are also close to these high scores.
The same-person and same-sex twin distributions lie largely above the dashed threshold. For this Twins Days figure, the threshold was calibrated to FMR = 0.0001 on unrelated mugshot pairs; the twin FMR is much higher.
Age Effects
In its technical summary, the report describes lower similarity scores among older different-identity twin pairs in both datasets. Its later per-algorithm discussion says most algorithms show little or no age-related decline on immigration-related images, so this is not a uniform trend.
Twins Days dataset categorizes twins into four age groups: 0-19, 20-39, 40-59, and 60+. Analysis shows lower similarity scores in the oldest age group for both fraternal and identical same-sex twins compared to the youngest age group.
Immigration-related dataset includes only same-sex twins, without labels for identical versus fraternal type, divided into three age groups: 0-19, 20-39, and 40-59. The summary describes scores for the 0-19 group as closer to those of same-identity comparisons and those for the 40-59 group as lower; the later per-algorithm results do not show this decline for most algorithms.
Even where older-age similarity scores decline, many different-identity twin pairs remain above the algorithm thresholds and are mistaken for the same person. The report tests whether the two people can be distinguished; it does not ask algorithms to classify a pair as identical or fraternal twins.
Longitudinal Effects
Across the submissions examined in this report, better general mugshot-verification performance did not bring a corresponding decline in false matches between same-sex twins. The observation concerns the tested algorithms and Twins Days data, not every later system.
Conclusion
Considering NIST's Face Recognition Vendor Test (FRVT) performance and its ability to distinguish between identical and fraternal twins, the algorithms tested in this report face significant challenges, particularly in distinguishing genetically similar identical twins.
Despite lower genetic similarity and potential greater facial differences among fraternal twins, algorithms still struggle to differentiate them, which could pose serious risks in high-accuracy identity checks. The consequences depend on the application and its safeguards.
For future improvements, the report proposes testing whether retaining higher-resolution facial detail helps distinguish twins.
Using Higher Resolution Images
Higher resolution images can provide more detailed facial skin features, such as skin texture and pore patterns, which are unique to individuals and could help distinguish even genetically identical twins.
Future work could improve image capture quality. The report notes historical skin-texture work using compliant frontal portraits with at least 120 pixels between the eyes, while later Twins Days evaluation was planned with at least 500 pixels. These are report-specific settings, not a universal pass criterion. Additionally, mainstream neural network models often downsample input images to improve processing speed, but this loses crucial detail information, affecting recognition accuracy (especially in highly similar cases like twins).
These are directions for future research. We hope later FRVT evaluations can show whether twin recognition improves under comparable operating conditions.