Most neural systems are adapted by optimization of a performance index, typically the minimization of a \cost function", based on a nite database (a training set) of N noisy examples derived from the target system. However, there is always the hidden

Generalization: The Hidden Agenda of Learning by Jan Larsen and Lars Kai Hansen Department of Mathematical Modelling Technical University of Denmark emails: jl,lkhansen@imm.dtu.dk

Most neural systems are adapted by optimization of a performance index, typically the minimization of a\cost function", based on a nite database (a training set) of N noisy examples derived from the target system. However, there is always the hidden agenda that the model should perform well, not only on the training set, but on the much larger set of future inputs to the system. Reading for your nals you solve previous years tests, but you know very well that if you then test yourself on last years test the result will be biased{ too optimistic! Only a test on a fresh data set, a test that was put aside before you started reading, will give the you a reliable prediction of the nal performance. Doing well on unseen data may at rst seem unattainable, but the ability to generalize in very complex environments is nevertheless one of the most striking properties of neural systems, and indeed one of the reasons that neural networks have shown useful in practical applications. As an example: in 10] a neural network system for inspection of handwritten digits was able to classify 99.98% correct after training on a data base of 7291 digits, and classify 95% correct on an additional test set of 2007 digits. When using a super- exible model family, like neural networks, which in principle can model arbitrarily complex systems, over t is a major concern, which nds expression in the ubiquitous bias-variance dilemma 4]. The generalization ability of an adaptive system is the quantitative measure of performance on a hypothetical in nite test set. While this quantity cannot be accessed directly, algebraic asymptotic estimates of generalization, valid for large training sets (N ! 1), can be derived 1], 2], 9], 12], 13], 14]. Such asymptotic results were earlier derived for supervised learning however, it was recently shown that generalization ability for unsupervised learning machines (e.g., principal component analysis and clustering schemes) can be analyzed in a similar framework 7]. If su cient computational capacity is available, empirical resampling schemes can be invoked. The two basic resampling strategies are cross-validation and bootstrap. Crossvalidation 3], 15] is based on a random division of the database into disjunct training and validation sets. The procedure can be repeated, leading to more accurate results at the price of increased computation. The so-called leave-one-out cross-validation is based on using only a single example in the test set, and typically resampling N times. Approximative techniques, by which the computational overhead in leave-one-out is signi cantly reduced, has been reported 8], 13]. Bootstrap, invented by Efron 6], is based on resampling with replacement. Bootstrap produces pseudo training sets of size N, hence, simulates training set uctuati

ons at the full sample size, and was applied to control of over t in a number of investigations 5], 17], 18]. Optimization of the neural network architecture may lead to better generalization ability and preferably lower computational burden. optimizing the network architecture is Appears in J.-N. Hwang, S.Y. Kung, M. Niranjan& J.C. Principe (eds.) The Past, Present, and Future of Neural Networks for Signal Processing, IEEE Signal Processing Magazine, pp. 43{45, Nov. 1997.