Natural gradient matrix momentum

Abstract

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.

Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: ©1999 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Uncontrolled Keywords: Natural gradient learning,on-line learning,Fisher information matrix,matrix momentum algorithm,two-layer network
Last Modified: 02 Jan 2024 08:25
Date Deposited: 04 Aug 2009 14:21
Full Text Link:
Related URLs: http://ieeexplo ... &isnumber=17760 (Publisher URL)
PURE Output Type: Chapter
Published Date: 1999-09-07
Authors: Scarpetta, Silvia
Rattray, Magnus
Saad, David (ORCID Profile 0000-0001-9821-2623)

Download

[img]

Version: Published Version


Export / Share Citation


Statistics

Additional statistics for this record