A multimodal-Siamese Neural Network (mSNN) for person verification using signatures and EEG


Signatures have long been considered to be one of the most accepted and practical means of user verification, despite being vulnerable to skilled forgers. In contrast, EEG signals have more recently been shown to be more difficult to replicate, and to provide better biometric information in response to known a stimulus. In this paper, we propose combining these two biometric traits using a multimodal Siamese Neural Network (mSNN) for improved user verification. The proposed mSNN network learns discriminative temporal and spatial features from the EEG signals using an EEG encoder and from the offline signatures using an image encoder. Features of the two encoders are fused into a common feature space for further processing. A Siamese network then employs a distance metric based on the similarity and dissimilarity of the input features to produce the verification results. The proposed model is evaluated on a dataset of 70 users, comprised of 1400 unique samples. The novel mSNN model achieves a 98.57% classification accuracy with a 99.29% True Positive Rate (TPR) and False Acceptance Rate (FAR) of 2.14%, outperforming the current state-of-the-art by 12.86% (in absolute terms). This proposed network architecture may also be applicable to the fusion of other neurological data sources to build robust biometric verification or diagnostic systems with limited data size.

Publication DOI: https://doi.org/10.1016/j.inffus.2021.01.004
Divisions: College of Business and Social Sciences > Aston Business School
College of Business and Social Sciences > Aston Business School > Operations & Information Management
Additional Information: © 2021, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ Funding Information: Prof Chang’s research is partly supported by VC Research (number: VCR 0000050 ). Dr. Scheme’s research is partly supported by the New Brunswick Innovation
Uncontrolled Keywords: CNN,EEG,LSTM,Multimodal,Siamese Neural Network,User verification,Software,Signal Processing,Information Systems,Hardware and Architecture
Publication ISSN: 1566-2535
Full Text Link:
Related URLs: http://www.scop ... tnerID=8YFLogxK (Scopus URL)
https://www.sci ... 0105?via%3Dihub (Publisher URL)
PURE Output Type: Article
Published Date: 2021-07-01
Published Online Date: 2021-01-19
Accepted Date: 2021-01-10
Authors: Chakladar, Debashis Das
Kumar, Pradeep
Roy, Partha Pratim
Dogra, Debi Prosad
Scheme, Erik
Chang, Victor (ORCID Profile 0000-0002-8012-5852)

Export / Share Citation


Additional statistics for this record