Natural gradient matrix momentum

Scarpetta, Silvia, Rattray, Magnus and Saad, David (1999). Natural gradient matrix momentum. IN: Ninth International Conference on Artificial Neural Networks, ICANN 99. Conference Publication, 1 . Edinburgh UK: IEEE.

Abstract

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.

Divisions: Engineering & Applied Sciences > Mathematics
Engineering & Applied Sciences > Systems analytics research institute (SARI)
Additional Information: ©1999 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Uncontrolled Keywords: Natural gradient learning,on-line learning,Fisher information matrix,matrix momentum algorithm,two-layer network
Full Text Link:
Related URLs: http://ieeexplo ... &isnumber=17760 (Publisher URL)
Published Date: 1999-09-07
Authors: Scarpetta, Silvia
Rattray, Magnus
Saad, David ( 0000-0001-9821-2623)

Download

[img]

Version: Published Version


Export / Share Citation


Statistics

Additional statistics for this record