The dynamics of matrix momentum


We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: The original publication is available at
ISBN: 3540762639
Last Modified: 29 Nov 2023 13:43
Date Deposited: 22 Sep 2009 13:28
Full Text Link: 10.1007/978-1-4471-1599-1_24
Related URLs: https://link.sp ... -4471-1599-1_24 (Publisher URL)
PURE Output Type: Chapter
Published Date: 1998-09-01
Authors: Rattray, Magnus
Saad, David



Version: Accepted Version

Export / Share Citation


Additional statistics for this record