DescriptionMost statistical estimation routines are, in some way, based on minimizing or maximizing a certain objective function, for instance least squares or Maximum Likelihood. These techniques hit a natural limit when the model possesses more parameters than data points, since invidual parameteters lose at this point their identifiability and standard estimation methods break down; for instance due to singularity of matrices that need to be inverted. While, from a traditional point of view, one may consider this problem (often referred to as the `p>n' problem) as a rather hypothetical consideration, modern data sets frequently show exactly this feature. For instance, genetic microarray data typically involve observations on some dozens of cells (observations) but thousands of genes (parameters). Hence, any statistical model dealing with such data is per se over-parametrized. Regularization is referred to as the process of dealing with overparametrized models by including a `penalty' term which enforces constraints on the parameters, making the estimation step feasible again. One of the most famous representatives of regularization methods is the `LASSO', which minimizes a LS criterrion with absolute-norm penalty,
(see also Reference 1. and 2. below), where λ is referred to as regularization (or penalty) parameter. The very useful feature of the LASSO is that it `shrinks' all parameters to 0 which are
not really needed, hence carrying out implicit variable selection, enabling a sparse representation of the relationship
between predictors and responses. Regularization appears in many other forms and facets, and can be considered as one of the current `hot' topics in Statistics and Machine Learning (Ref 3.). For instance, regularization is used in nonparametric regression where it enforces an adequate degree of smoothness of the regression function (see Ref. 4), or the analysis of multivariate time series (see Ref. 4).
Prerequisites
Resources
|
email: jochen.einbeck "at" durham.ac.uk