Regularized least squares (RLS) is a family of methods for solving the
least-squares
The method of least squares is a mathematical optimization technique that aims to determine the best fit function by minimizing the sum of the squares of the differences between the observed values and the predicted values of the model. The me ...
problem while using
regularization
Regularization may refer to:
* Regularization (linguistics)
* Regularization (mathematics)
* Regularization (physics)
* Regularization (solid modeling)
* Regularization Law, an Israeli law intended to retroactively legalize settlements
See also ...
to further constrain the resulting solution.
RLS is used for two main reasons. The first comes up when the number of variables in the linear system exceeds the number of observations. In such settings, the
ordinary least-squares problem is
ill-posed and is therefore impossible to fit because the associated optimization problem has infinitely many solutions. RLS allows the introduction of further constraints that uniquely determine the solution.
The second reason for using RLS arises when the learned model suffers from poor
generalization
A generalization is a form of abstraction whereby common properties of specific instances are formulated as general concepts or claims. Generalizations posit the existence of a domain or set of elements, as well as one or more common characteri ...
. RLS can be used in such cases to improve the generalizability of the model by constraining it at training time. This constraint can either force the solution to be "sparse" in some way or to reflect other prior knowledge about the problem such as information about correlations between features. A
Bayesian
Thomas Bayes ( ; c. 1701 – 1761) was an English statistician, philosopher, and Presbyterian minister.
Bayesian ( or ) may be either any of a range of concepts and approaches that relate to statistical methods based on Bayes' theorem
Bayes ...
understanding of this can be reached by showing that RLS methods are often equivalent to
priors
Prior (or prioress) is an ecclesiastical title for a superior in some religious orders. The word is derived from the Latin for "earlier" or "first". Its earlier generic usage referred to any monastic superior. In abbeys, a prior would be lowe ...
on the solution to the least-squares problem.
General formulation
Consider a learning setting given by a probabilistic space
,
. Let
denote a training set of
pairs i.i.d. with respect to the joint distribution
. Let