The theory of on-line learning: a statistical physics approach


In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: The original publication is available at
Event Title: Studies in Classification, Data Analysis and Knowledge Organization
Event Type: Other
Event Dates: 2003-01-01 - 2003-01-01
Uncontrolled Keywords: on-line learning,neural networks,statistical physics,natural gradient descent,matrix-momentum,repetition
ISBN: 9783540441830
Last Modified: 26 Apr 2024 07:23
Date Deposited: 14 Sep 2009 13:11
Full Text Link:
Related URLs: http://www.spri ... detailsPage=toc (Publisher URL)
PURE Output Type: Chapter
Published Date: 2003
Authors: Saad, David (ORCID Profile 0000-0001-9821-2623)



Version: Accepted Version

Export / Share Citation


Additional statistics for this record