MOBIO Face and Speaker Verification Evaluation
The MOBIO consortium is organizing a contest for the next International Conference on Pattern Recognition (ICPR) to be held in 2010 at Istanbul on August 23.
MOBIO (Mobile Biometry) is a European Project focusing on biometric person recognition from portable devices such as mobile phones. The project is focusing on multiple aspects of biometric person recognition, ranging from research to development, and more particularly on robust-to-illumination face localization and verification, speaker verification in noisy environments, bi-modal fusion, unsupervised model adaptation through time and scalability.
The project consortium is collecting a large bi-modal (audio/visual) database recorded from mobile phones across multiple sites in Europe. We plan to make publicly available this new audio-visual database of synchronized face and speech samples collected on mobile devices. The initial distribution of this database will be achieved through the organization of a contest for the next International Conference on Pattern Recognition (ICPR) in August 2010.
This contest will focus on evaluating the performance of uni-modal face and speaker verification techniques in the context of a mobile environment, thus offering challenging recording conditions (adverse illumination, noisy background).
The availability of common benchmark databases, together with evaluation protocols has been partly responsible for the significant gains made in biometric person recognition in recent years. We believe that such evaluations should be continued. Databases and more importantly unbiased evaluation mechanisms should be spread across the scientific community making it possible for scientists to evaluate their progress.
Pre-registration:
Pre-registration is closed. The list of potential participants is available at the following Doodle
Organizers:
Sébastien Marcel and Chris McCool
Idiap Research Institute, Switzerland (CH)
Timo Ahonen
University of Oulu, Finland (FI)
Honza Cernocky
Brno University of Technology, Czech Republic (CZ)
Timeline (revised on Nov 30 2009):
Nov 30 2009 : Deadline for pre-registration
Dec 1 2009 : Database available (until Jan 27)
Here is a description of the procedure to download and use the MOBIO database :
Downloading the database
The database will be available to participants ONLY. To become a participant, you have to :
1. download, fill, and sign the End User License Agreement (EULA) below :
Upon reception of the signed EULA, an account to a private section on the MOBIO website will then be created for you at https://www.mobioproject.org/icpr-download where you will be able to download the database. You will also be automatically added on the mailing list icpr-eval@mobioproject.org to receive instructions related to the competition.
Using the database
Instructions (and tools) about how to use precisely (protocol files) the database will also be available in your private section.
The database is split into 3 disctinct sets : one for training, one for development and one for testing. For a short description of the MOBIO database provided for the ICPR 2010 competition please refer to this document :
MOBIO Database for the ICPR 2010 Face and Speech Competition,
Chris McCool and Sébastien Marcel, McCool_Idiap-Com-02-2009,
URL : http://publications.idiap.ch/index.php/publications/show/1757.
Important note about the evaluation
The evaluation will be organized in 2 steps :
Step 1 : initial results should be provided using the training and development splits together with a form describing your algorithms. Therefore, only those 2 splits will be available for download from Dec 1 2009 to Feb 1 2010.
Step 2 : final results should be provided using the test split. The test split will be provided on Feb 1 2010 ONLY to participants providing initial results with their forms or the account will be closed.
January 15 2010 : Form to describe algorithms available
The forms are now available in the private section https://www.mobioproject.org/icpr-download/download/forms .
January 25 2010 : Deadline for ICPR regular paper submission
Please note that participants to the evaluation can submit, if they wish, a regular paper on the algorithm(s) used for the evaluation. This submission is independant from the evaluation and from the final overview paper of the contest.
January 27 2010 : Database no longer available for new participants.
February 1 2010 : Deadline for the submission of the initial results by the participants
You should send the results (scores + form) as a compressed folder (zip or tar.gz) to the organizers at the following mailing list: icpr-competition@mobioproject.org NOT icpr-eval@mobioproject.org
The naming convention for the file is the following: <SITE>_<MODALITY>.tar.gz
where <SITE> is the accronym of your institution (IDIAP, UOULU, BUT, MIT, CMU, ...)
and <MODALITY> is either A for Audio, V for Video and eventually F for Fusion.
If you provide several systems (no more than 2 please) then add a simple number.
Examples:
- MIT provided the results of a face verification system: MIT_V.zip
- BUT provided the results of 2 speaker verification systems: BUT_A1.tar.gz and BUT_A2.tar.gz
- UNIS provided the results of a fusion system: UNIS_F.zip as well as UNIS_A.zip and UNIS_V.zip
March 8 2010 : Deadline for the submission of the final results by the participants
April 30 2010 : Deadline for the overview paper of the contest
August 23 2010 : ICPR 2010 at Istanbul
The performance evaluation methodology:
Biometric person verification consists of confirming (or denying) a clients claim supported by captured biometric data such as their face image or their voice recording. In a face or speech verification scenario, the claimant either provides their true identity or he/she is trying to fool the system (he/she is then referred to as an impostor). More precisely, verification is achieved by computing a score, a single scalar (similarity measure, likelihood ratio, classifier output, ...) assessing the "similarity" between the biometric data and the reference biometric model (also called client model) associated to the claimed identity. Then a decision has to be made on whether the claimant is a true client or an impostor by comparing the verification score to a threshold.
Two kinds of error can occur: either the true claimant is rejected (false rejection), or an impostor is accepted (false acceptance). Errors are typically presented in a 2D plot (ROC or DET) which shows the False Acceptance Rate (FAR) as a function of the False Rejection Rate (FRR) for different values of the threshold. Classical ROC/DET curves can be in some cases misleading and alternatives such as Expected Performance Curve (EPC) will be also considered.
To obtain unbiased and comparative results, and thus to assess the quality of the results, an experimental
protocol should be provided. An experimental protocol describes precisely the partitioning of the database into a world model set, a training set, a development set and a test set. The world model set should be used to compute any background model (PCA or LDA projection matrices, prior statistical model, ...) required for further training and testing. The training set is the set of client samples to be used to enroll client models. The development set is the set of clients and impostors used for tuning the decision threshold. The test set is the set of clients and impostors used for evaluating the performance using the previous decision threshold.
Home
Project
Publications
