I was struggling with Ridge Regression for a long time because I don’t understand the purpose of it. Until one day I wake up in the morning, I finally realize that Ridge Regression is Normal equation with regularization.
The equation likes :
Moreover, lambda in this equation does the same function in the last part of the post of Multiple variables linear regression
With bigger lambda, the model becomes more general.
With small lambda, the model may overfitting.
def RidgeRegression(X,mtip,lam=0.001): #according to equation W = (X.T*X+eye(shape(X))*lam ).I*(X.T*mtip.T) return W
You can replace this function with linear regression and see how it goes. Also, Ridge Regression avoid not invertible problems in linear regression. On the other hands, because of regularization, some variables will be eliminated to avoid overfitting. Ridge Regression also be think as shrinkage methods like PCA, which may explain later.