Condition estimation for regression and feature selection

[摘要] Regression and feature selection require the minimisation of parallel to X beta - y parallel to(2) with respect to beta, where X is an element of R-nxp, n < p in feature selection and n >= p in regression. The vector beta contains the coefficients of the basis functions in regression, and the weights of the features in feature selection. This paper considers the stability of beta, as measured by the ratio of its relative error with respect to the relative error in y, and it is shown that the condition number kappa(X) of X is not a good measure of this stability. In particular, a large value of kappa(X) may lead to incorrect conclusions about the stability of beta and it may be thought regularisation must be applied to the normal equation (XX)-X-T beta = X(T)y if kappa(X) >> 1, but its application may lead to a large error in beta. It is shown in this paper that (a) the presence of noise in y or the condition kappa(X) >> 1 do not imply that regularisation must be applied to the normal equation, and (b) the condition kappa(X) >> 1 does not imply that a small relative error in y yields a large relative error in beta. These disadvantages of kappa(X) lead to the effective condition number eta(X, y), which provides a better measure of the stability of beta due to a perturbation in y, but it is difficult to compute it reliably in some circumstances. Regularisation requires that a constraint be imposed on the solution of the normal equation, and it is shown that a constraint on parallel to beta parallel to(1) can be interpreted in terms of the column sums of X, and that a constraint on parallel to beta parallel to(2) can be interpreted in terms of the singular value decomposition of X. The paper contains several examples that illustrate the theoretical results. (C) 2019 Elsevier B.V. All rights reserved.

[发布日期] 2020-08-01

[效力级别]  Proceedings Paper

[关键词] Condition estimation;Regression;Tikhonov regularisation;Feature selection