The role of biases in on-line learning of two-layer networks


The influence of biases on the learning dynamics of a two-layer neural network, a normalized soft-committee machine, is studied for on-line gradient descent learning. Within a statistical mechanics framework, numerical studies show that the inclusion of adjustable biases dramatically alters the learning dynamics found previously. The symmetric phase which has often been predominant in the original model all but disappears for a non-degenerate bias task. The extended model furthermore exhibits a much richer dynamical behavior, e.g. attractive suboptimal symmetric phases even for realizable cases and noiseless data.

Publication DOI:
Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: Copyright of the American Physical Society
Uncontrolled Keywords: learning dynamics,two-layer neural network,soft-committee machine,on-line gradient descent learning
Publication ISSN: 1550-2376
Last Modified: 02 Jan 2024 08:04
Date Deposited: 11 Mar 2019 17:29
Full Text Link:
Related URLs: http://prola.ap ... /v57/i3/p3265_1 (Publisher URL)
PURE Output Type: Article
Published Date: 1998-03
Authors: West, Ansgar H.L.
Saad, David (ORCID Profile 0000-0001-9821-2623)



Version: Accepted Version

| Preview

Export / Share Citation


Additional statistics for this record