Comparative Study of Least Squares, Ridge, Lasso and Principal Component Regression Algorithms

Abstract

Least squares and other traditional statistical methods are widely used for fitting predictive analytic models. However, the methods have a number of limitations that can lower the predictive accuracy of the models. The study considered a typical dataset with two main objectives. First, to fit four predictive models by applying least squares and three trending machine learning algorithms - ridge regression, lasso regression and principal component regression and second, compare their performance. The methods were assessed using 5-fold cross validation technique. The result revealed that principal component regression outperformed the others with minimal mean square error of -7.55E+21. This suggested that principal component regression would perform optimally for building predictive model for the related dataset.