Prior knowledge and the creation of virtual examples for RBF networks

1 to appear in the Proceedings of the IEEE Workshop on Neural Networks for Signal Processing, September 1995, Cambridge, MA.
case he argues that the \hints" can be represented by new examples, generated from the existing data set by applying transformations that are known to leave the function to be learned invariant. This approach can be applied independently of the learning technique that is used: the learning technique remains the same, and the data set is augmented with new, \virtual" examples. An alternative approach consists in leaving the data set unaltered, but to use the prior knowledge to modify an existing technique to ensure that the approximated function has the desired invariance properties. This approach is clearly related to the creation of virtual examples, but it is not obvious that provides the same result. In the following we apply this alternative technique to the case in which the prior knowledge consists in knowing that a function is radially symmetric. The approximation technique we consider is Radial Basis Functions, because it can be derived as the solution of a functional minimization problem, in which prior knowledge about the smoothness of the function is already used. The presence of a functional to be minimized makes easy to introduce the additional prior knowledge as a constraint over the domain of the functional, and will let us derive an analytical solution, that is as simple as in the Radial Basis Functions case. Interestingly enough, the solution derived in this way is exactly the same that is obtained if the prior knowledge is used to create virtual examples, showing that the creation of virtual examples is the \right" thing to do, and providing another step in a rigorous mathematical analysis of the technique of virtual examples. Before describing these results we rst brie y review the regularization theory approach to function approximation. Suppose that the set D = f(xi ; yi ) 2 Rd RgN is a random, noisy sample i=1 of some multivariate function h. The problem of recovering the function h from the data D is ill-posed, and can be formulated in the framework of regularization theory (Tikhonov, 1963; Wahba, 1990; Poggio and Girosi, 1990). In this framework the solution is found by minimizing a functional of the form:
1 Introduction
Lack of examples is very often responsible for poor performances of learning algorithms. In many cases it is di cult, if not impossible, to collect additional data, leaving us with unsatisfactory solutions. However, it is often the case that not only the examples but also some prior knowledge on the learning target are available. Examples of prior knowledge are smoothness, invariance with respect to transformation groups (such as rotations or re ections) or information about time and/or space scale. Most of the existing learning schemes do not make use of prior knowledge, and therefore provide suboptimal solutions that do not fully exploit the amount of information available. One major contribution to this topic has been given by Abu-Mostafa (1993) who developed a methodology for integrating di erent kinds of \hints" (prior knowledge) into the usual learning-from-example procedure, and related them to the well-known concept of VC-dimension (Vapnik, 1982). AbuMostafa considers, among other things, the case in which the function that has to be learned is invariant with respect to certain transformations. In this
PRIOR KNOWLEDGE AND THE CREATION OF \VIRTUAL" EXAMPLES FOR RBF NETWORKS1 Federico Girosi and Nicholas Tung Chan
Center for Biological and Computational Learning and Arti cial Intelligence Laboratory Massachusetts Institute of Technology Cambridge MA 02139 USA
H f] =
N X i=1
2 Regularization Theory and RBF
(f (xi ) ? yi )2 +
Hale Waihona Puke f] :(1)where is a positive number that is usually called the regularization parameter and f ] is a cost functional that constrains the space of possible solutions according to some form of prior knowledge. The most common form of prior knowledge is smoothness, that, in words, ensures that if two inputs are close the two corresponding output are also close. We consider here a very general
Abstract We consider the problem of how to incorporate prior knowledge in supervised learning techniques. We set the problem in the framework of regularization theory, and consider the case in which we know that the approximated function has radial symmetry. The problem can be solved in two alternative ways: 1) use the invariance as a constraint in the regularization theory framework to derive a rotation invariant version of Radial Basis Functions; 2) use the radial symmetry to create new, \virtual" examples from a given data set. We show that these two apparently di erent methods of learning from \hints" (Abu-Mostafa, 1993) lead to exactly the same analytical solution.