Finite-size effects in on-line learning of multilayer neural networks


We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.

Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: Copyright of EDP Sciences
Uncontrolled Keywords: probability theory,stochastic processes,and statistics
Publication ISSN: 1286-4854
Last Modified: 27 Jun 2024 07:35
Date Deposited: 08 Jul 2009 12:14
Full Text Link:
Related URLs: http://iopscien ... ect=.iopscience (Publisher URL)
PURE Output Type: Article
Published Date: 1996-04
Authors: Barber, David
Saad, David (ORCID Profile 0000-0001-9821-2623)
Sollich, Peter



Version: Published Version

Export / Share Citation


Additional statistics for this record