Large deviation analysis of function sensitivity in random deep neural networks

Li, Bo and Saad, David (2020). Large deviation analysis of function sensitivity in random deep neural networks. Journal of Physics A: Mathematical and Theoretical, 53 (10),

Abstract

Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.

Publication DOI:	https://doi.org/10.1088/1751-8121/ab6a6f
Divisions:	College of Engineering & Physical Sciences College of Engineering & Physical Sciences > Systems analytics research institute (SARI) Aston University (General)
Additional Information:	© 2020 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Uncontrolled Keywords:	deep neural networks,function sensitivity,large deviation theory,path integral,Statistical and Nonlinear Physics,Statistics and Probability,Modelling and Simulation,Mathematical Physics,General Physics and Astronomy
Publication ISSN:	1751-8121
Last Modified:	31 Oct 2024 18:24
Date Deposited:	13 Jan 2020 11:31
Full Text Link:	https://arxiv.o ... /abs/1910.05769
Related URLs:	http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type:	Article
Published Date:	2020-02-20
Published Online Date:	2020-01-10
Accepted Date:	2020-01-09
Authors:	Li, Bo ( 0000-0001-9743-9447) Saad, David ( 0000-0001-9821-2623)