A Pass to Variable Selection

[摘要] Many regularized procedures produce sparse solution andtherefore are sometimes used for variable selection in linear regression.It has been showed that regularized procedures are more stable thansubset selection. Such procedures include LASSO, SCAD, and adaptiveLASSO, to name just a few. However, their performance dependscrucially on the tuning parameter selection. For the purpose ofprediction, popular methods for the tuning parameter selection includeCp, cross-validation, and generalized cross-validation. For the purposeof variable selection, the most popular method for the tuning parameterselection is BIC. The selection consistency of BIC for some regularizedprocedures have been shown. However, knowing degrees of freedom isrequired in the use of BIC. For many regularized procedures, such asthose for graphical models and clustering algorithms, the formulae fordegrees of freedom do not exist.