In
functional analysis
Functional analysis is a branch of mathematical analysis, the core of which is formed by the study of vector spaces endowed with some kind of limit-related structure (e.g. inner product, norm, topology, etc.) and the linear functions defi ...
(a branch of
mathematics
Mathematics is an area of knowledge that includes the topics of numbers, formulas and related structures, shapes and the spaces in which they are contained, and quantities and their changes. These topics are represented in modern mathematics ...
), a reproducing kernel Hilbert space (RKHS) is a
Hilbert space
In mathematics, Hilbert spaces (named after David Hilbert) allow generalizing the methods of linear algebra and calculus from (finite-dimensional) Euclidean vector spaces to spaces that may be infinite-dimensional. Hilbert spaces arise natural ...
of functions in which point evaluation is a continuous linear
functional
Functional may refer to:
* Movements in architecture:
** Functionalism (architecture)
** Form follows function
* Functional group, combination of atoms within molecules
* Medical conditions without currently visible organic basis:
** Functional sy ...
. Roughly speaking, this means that if two functions
and
in the RKHS are close in norm, i.e.,
is small, then
and
are also pointwise close, i.e.,
is small for all
. The converse does not need to be true. Informally, this can be shown by looking at the
supremum norm
In mathematical analysis, the uniform norm (or ) assigns to real- or complex-valued bounded functions defined on a set the non-negative number
:\, f\, _\infty = \, f\, _ = \sup\left\.
This norm is also called the , the , the , or, when th ...
: the sequence of functions
converges pointwise, but do not converge
uniformly i.e. do not converge with respect to the supremum norm (note that this is not a counterexample because the supremum norm does not arise from any
inner product
In mathematics, an inner product space (or, rarely, a Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation called an inner product. The inner product of two vectors in the space is a scalar, often ...
due to not satisfying the
parallelogram law).
It is not entirely straightforward to construct a Hilbert space of functions which is not an RKHS. Some examples, however, have been found.
Note that
''L''2 spaces are not Hilbert spaces of functions (and hence not RKHSs), but rather Hilbert spaces of equivalence classes of functions (for example, the functions
and
defined by
and
are equivalent in ''L''
2). However, there are RKHSs in which the norm is an ''L''
2-norm, such as the space of band-limited functions (see the example below).
An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every
in the set on which the functions are defined, "evaluation at
" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists if and only if every evaluation functional is continuous.
The reproducing kernel was first introduced in the 1907 work of
Stanisław Zaremba concerning
boundary value problem
In mathematics, in the field of differential equations, a boundary value problem is a differential equation together with a set of additional constraints, called the boundary conditions. A solution to a boundary value problem is a solution to t ...
s for
harmonic
A harmonic is a wave with a frequency that is a positive integer multiple of the ''fundamental frequency'', the frequency of the original periodic signal, such as a sinusoidal wave. The original signal is also called the ''1st harmonic'', t ...
and
biharmonic functions.
James Mercer simultaneously examined
functions which satisfy the reproducing property in the theory of
integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of
Gábor Szegő
Gábor Szegő () (January 20, 1895 – August 7, 1985) was a Hungarian-American mathematician. He was one of the foremost mathematical analysts of his generation and made fundamental contributions to the theory of orthogonal polynomials and ...
,
Stefan Bergman
Stefan Bergman (5 May 1895 – 6 June 1977) was a Congress Poland-born American mathematician whose primary work was in complex analysis. His name is also written Bergmann; he dropped the second "n" when he came to the U. S. He is best known for t ...
, and
Salomon Bochner
Salomon Bochner (20 August 1899 – 2 May 1982) was an Austrian mathematician, known for work in mathematical analysis, probability theory and differential geometry.
Life
He was born into a Jewish family in Podgórze (near Kraków), then ...
. The subject was eventually systematically developed in the early 1950s by
Nachman Aronszajn
Nachman Aronszajn (26 July 1907 – 5 February 1980) was a Polish American mathematician. Aronszajn's main field of study was mathematical analysis, where he systematically developed the concept of reproducing kernel Hilbert space. He also cont ...
and Stefan Bergman.
These spaces have wide applications, including
complex analysis
Complex analysis, traditionally known as the theory of functions of a complex variable, is the branch of mathematical analysis that investigates Function (mathematics), functions of complex numbers. It is helpful in many branches of mathemati ...
,
harmonic analysis
Harmonic analysis is a branch of mathematics concerned with the representation of functions or signals as the superposition of basic waves, and the study of and generalization of the notions of Fourier series and Fourier transforms (i.e. an ex ...
, and
quantum mechanics
Quantum mechanics is a fundamental theory in physics that provides a description of the physical properties of nature at the scale of atoms and subatomic particles. It is the foundation of all quantum physics including quantum chemistry, ...
. Reproducing kernel Hilbert spaces are particularly important in the field of
statistical learning theory
Statistical learning theory is a framework for machine learning drawing from the fields of statistics and functional analysis. Statistical learning theory deals with the statistical inference problem of finding a predictive function based on dat ...
because of the celebrated
representer theorem
For computer science, in statistical learning theory, a representer theorem is any of several related results stating that a minimizer f^ of a regularized empirical risk functional defined over a reproducing kernel Hilbert space can be represe ...
which states that every function in an RKHS that minimises an empirical risk functional can be written as a
linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the
empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem.
For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of
analytic functions.
Definition
Let
be an arbitrary
set
Set, The Set, SET or SETS may refer to:
Science, technology, and mathematics Mathematics
*Set (mathematics), a collection of elements
*Category of sets, the category whose objects and morphisms are sets and total functions, respectively
Electro ...
and
a
Hilbert space
In mathematics, Hilbert spaces (named after David Hilbert) allow generalizing the methods of linear algebra and calculus from (finite-dimensional) Euclidean vector spaces to spaces that may be infinite-dimensional. Hilbert spaces arise natural ...
of
real-valued function
In mathematics, a real-valued function is a function whose values are real numbers. In other words, it is a function that assigns a real number to each member of its domain.
Real-valued functions of a real variable (commonly called ''real f ...
s on
, equipped with pointwise addition and pointwise scalar multiplication. The
evaluation
Evaluation is a
systematic determination and assessment of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization, program, design, project or any other intervention or initiative to ...
functional over the Hilbert space of functions
is a linear functional that evaluates each function at a point
,
:
We say that ''H'' is a reproducing kernel Hilbert space if, for all
in
,
is
continuous
Continuity or continuous may refer to:
Mathematics
* Continuity (mathematics), the opposing concept to discreteness; common examples include
** Continuous probability distribution or random variable in probability and statistics
** Continuous g ...
at every
in
or, equivalently, if
is a
bounded operator
In functional analysis and operator theory, a bounded linear operator is a linear transformation L : X \to Y between topological vector spaces (TVSs) X and Y that maps bounded subsets of X to bounded subsets of Y.
If X and Y are normed vector ...
on
, i.e. there exists some
such that
Although
is assumed for all
, it might still be the case that
.
While property () is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in
at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of
with a function
in
. This function is the so-called reproducing kernel for the Hilbert space
from which the RKHS takes its name. More formally, the
Riesz representation theorem
:''This article describes a theorem concerning the dual of a Hilbert space. For the theorems relating linear functionals to Measure (mathematics), measures, see Riesz–Markov–Kakutani representation theorem.''
The Riesz representation theorem, ...
implies that for all
in
there exists a unique element
of
with the reproducing property,
Since
is itself a function defined on
with values in the field
(or
in the case of complex Hilbert spaces) and as
is in
we have that
:
where
is the element in
associated to
.
This allows us to define the reproducing kernel of
as a function
by
:
From this definition it is easy to see that
(or
in the complex case) is both symmetric (resp. conjugate symmetric) and
positive definite In mathematics, positive definiteness is a property of any object to which a bilinear form or a sesquilinear form may be naturally associated, which is positive-definite. See, in particular:
* Positive-definite bilinear form
* Positive-definite fu ...
, i.e.
:
for every
The Moore–Aronszajn theorem (see below) is a sort of converse to this: if a function
satisfies these conditions then there is a Hilbert space of functions on
for which it is a reproducing kernel.
Example
The space of
bandlimited
Bandlimiting is the limiting of a signal's frequency domain representation or spectral density to zero above a certain finite frequency.
A band-limited signal is one whose Fourier transform or spectral density has bounded support.
A bandli ...
continuous function
In mathematics, a continuous function is a function such that a continuous variation (that is a change without jump) of the argument induces a continuous variation of the value of the function. This means that there are no abrupt changes in val ...
s
is a RKHS, as we now show. Formally, fix some
cutoff frequency
In physics and electrical engineering, a cutoff frequency, corner frequency, or break frequency is a boundary in a system's frequency response at which energy flowing through the system begins to be reduced ( attenuated or reflected) rather tha ...