The dynamics of matrix momentum

Rattray, Magnus and Saad, David (1998). The dynamics of matrix momentum. IN: Proceedings of the 8th International Conference on Artificial Neural Networks. Niklasson, Lars F.; Boden, Mikael B. and Ziemke, Tom (eds) Springer.

Abstract

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Publication DOI: https://doi.org/10.1007/978-1-4471-1599-1_24
Divisions: Engineering & Applied Sciences > Mathematics
Engineering & Applied Sciences > Systems analytics research institute (SARI)
Additional Information: The original publication is available at www.springerlink.com
Uncontrolled Keywords: matrix momentum,statistical mechanics,asymptotic performance,matrix inversion,Hessian
Full Text Link:
Related URLs: https://link.sp ... -4471-1599-1_24 (Publisher URL)
Published Date: 1998-09-01
Authors: Rattray, Magnus
Saad, David ( 0000-0001-9821-2623)

Download

[img]

Version: Accepted Version


Export / Share Citation


Statistics

Additional statistics for this record