Minimum description length, regularisation and multi-modal data


Conventional feed forward Neural Networks have used the sum-of-squares cost function for training. A new cost function is presented here with a description length interpretation based on Rissanen's Minimum Description Length principle. It is a heuristic that has a rough interpretation as the number of data points fit by the model. Not concerned with finding optimal descriptions, the cost function prefers to form minimum descriptions in a naive way for computational convenience. The cost function is called the Naive Description Length cost function. Finding minimum description models will be shown to be closely related to the identification of clusters in the data. As a consequence the minimum of this cost function approximates the most probable mode of the data rather than the sum-of-squares cost function that approximates the mean. The new cost function is shown to provide information about the structure of the data. This is done by inspecting the dependence of the error to the amount of regularisation. This structure provides a method of selecting regularisation parameters as an alternative or supplement to Bayesian methods. The new cost function is tested on a number of multi-valued problems such as a simple inverse kinematics problem. It is also tested on a number of classification and regression problems. The mode-seeking property of this cost function is shown to improve prediction in time series problems. Description length principles are used in a similar fashion to derive a regulariser to control network complexity.

Divisions: College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
Additional Information: Department: Engineering and Applied Science If you have discovered material in AURA which is unlawful e.g. breaches copyright, (either theirs or that of a third party) or any other law, including but not limited to those relating to patent, trademark, confidentiality, data protection, obscenity, defamation, libel, then please read our Takedown Policy and contact the service immediately.
Institution: Aston University
Uncontrolled Keywords: generalisation,minimum description length,regularisation,time series prediction
Last Modified: 08 Dec 2023 08:29
Date Deposited: 30 Jun 2010 11:00
Completed Date: 1995-11
Authors: Van Der Rest, John C. (ORCID Profile 0000-0002-3172-2714)


Export / Share Citation


Additional statistics for this record