What Is Lasso Regression?

Lasso regression is a statistical technique in machine learning that reduces overfitting by shrinking or reducing data toward a central mean. In machine learning (ML), regression analysis identifies how diverse data points impact each other. Training data is given to the ML algorithm (or model), which attempts to find the relationship between various factors. It then uses that relationship to predict the output for unknown inputs. However, as the number of factors increases, the input-output relationship becomes more complex and requires additional processing, or regularization, for improved accuracy. Lasso, or the least absolute shrinkage and selection operator, is a regularization technique in statistical models that causes a marginal decrease in training accuracy for an increase in generalizability.

Regularization is a technique that improves a machine learning model's accuracy when inferencing with unfamiliar real-world data. Machine learning models consider many parameters or features when making predictions. The model might produce inaccurate or inconsistent results depending on how they're configured. Data scientists attribute this to the model's bias and variance.

• Bias is the model's tendency to predict results different from the ground truth. It compares the predicted values the model generates with the absolute values.

• Variance is the ability to consistently make predictions within a narrow distributed range. It measures the model's sensitivity to changes in the training data.

An ideal model has low bias and low variance, which is nearly impossible to achieve. Regularization, however, can reduce the model's variance at a marginal cost of its bias. It introduces penalties to the model's loss function to achieve an optimal bias-variance tradeoff. The loss function, expressed as a mean squared error (MSE), measures and compares the model's prediction and expected outcomes. Predictor variables are the model's inputs to generate an outcome, while target variables are the expected results. Data scientists thus reduce the dependencies between predictor and target variables by applying loss function penalties.

Machine learning engineers use different regularization techniques to overcome overfitting.

L1 Regularization

L1 regularization is a technique for preventing the overfitting of linear regression models. It penalizes more important features and reduces the coefficients of the remaining ones to zero when calculating the MSE.

L2 Regularization

L2 regularization works similarly to L2 regularization in reducing the feature's weight when training a model. However, it shrinks the coefficients instead of reducing them to zero in the loss function.

Early stopping

Early stopping is a technique that halts model training before its validation accuracy decreases. In machine learning, the model first learns from a training dataset. Then, engineers assess the model's prediction accuracy by exposing it to a different validation dataset. Continuous training can increase validation error despite decreasing training error.

Data augmentation

Data augmentation is an approach where machine learning engineers artificially create new data samples from existing ones. This allows the model to overcome data scarcity issues, which might decrease its generalization capability with real-world data.

Dropout

Dropout is a regularizing technique that helps neural networks overcome co-dependencies among internal nodes. Regularization nodes are randomly selected and omitted from the training. In this setup, each layer has a random number of active nodes. These nodes must approximate the output without relying on the predictable input pattern from the original preceding nodes.

Weight decay

Weight decay regularizes neural networks to enable better generalization. It shares the same goal as L2 regularization. However, weight decay gradually decreases the weight at each training iteration instead of at the loss function.

Lasso regression is the application of L1 regularization in machine learning for linear models. Below, we compare lasso regression against similar regression methods.

Lasso regression vs. ridge regression

Ridge regression applies L2 regularization to reduce overfitting. When calculating the loss function, ridge regression sums up the squares of the regression coefficients. Meanwhile, lasso regression computes the absolute value of the coefficient. Both lasso and ridge regression regularize linear models. Ridge regression is more suitable for shrinking coefficients in complex models, while lasso regression helps in feature selection.

Lasso regression vs. linear regression

Linear regression is the primary form of regularization technique from which lasso regression is derived. A standard linear regression model tries to fit a line across distributed data points where the error between predictor and target variables is minimal. However, simple linear regression doesn't impose penalties on weights like lasso regression. Instead, it increases the weights deemed more important, increasing overfitting.

Lasso regression vs. elastic net

Elastic net regularization combines techniques used by Lasso regression and ridge regression. When calculating the loss function, it includes the absolute and squared values of the penalties. This allows data scientists to enable feature selection and address severe multicollinearity issues when training the model.

Lasso regression allows data scientists to omit unimportant features when training a machine learning model. The technique can also shrink specific features to reduce their significance when calculating the cost function. To understand lasso regression better, let's explore its basic formula.

Loss function = Root means squared error + Penalty.

Lasso regression's penalty is a function of the absolute coefficient value and the hyperparameter, λ. The goal is to achieve the minimum root mean squared error (RSME) by adjusting and finding the optimal λ value—a more significant value of λ results in a heavier penalty for the loss function. Meanwhile, replacing the absolute coefficient value with zero allows the model to train selected features.

These are steps that data scientists take to apply lasso regression in model training.

Conduct data analysis

Analyze the dataset for missing data, high correlation between dataset variables, and other discrepancies. Data scientists perform remedial measures such as feature scaling, normalization, and removing unwanted data to ensure quality datasets.

Prepare training and test datasets

Split the dataset into training and test data. Then, determine the target and predictor variables in the training dataset. A machine learning model uses the training data to analyze and learn the underlying pattern, which allows it to generalize more accurately. Meanwhile, the test data contains input data and the expected outcomes to evaluate the model's performance.

Implement lasso regression techniques

Create a lasso regression model and feed it with the training dataset. Then, the model runs several iterations of prediction to find a λ that produces the smallest root mean squared error (RMSE). During the process, the lasso regression algorithm reduces the model's complexity by replacing unimportant coefficients with zero.

Interpret the results

After the lasso regression model completes all predictions, you can evaluate different metrics to determine if the technique results in a simpler and more accurate model for your dataset.

Lasso regression is helpful in these scenarios.

Dimensionality reduction

Dimensionality reduction allows a machine learning model to learn to extract meaningful representation from complex, high-dimensional data. High-dimensional datasets consist of large numbers of features, such as image or video data. With lasso regression, you can create simpler, sparse models that retain essential information by removing unnecessary features.

Feature selection

Feature selection improves a model's performance by learning from important data and discarding irrelevant ones. When training the model, lasso regression can exclude unimportant features by replacing specific coefficients with zero. This allows the model to be more selective in learning, prioritizing valuable features over noise and redundant data.

Multicollinearity

Multicollinearity is a circumstance where two or more predictor variables are related. The predictor variables are the features in your dataset that you use to predict a target variable. When you have multicollinearity, the predictor variables are not only predictive of the target variable but also predictive of each other. You can use lasso regression as a measure of multicollinearity in your data.

Amazon SageMaker is a fully managed service you can use to prepare data and build, train, and deploy machine learning models for any use case with fully managed infrastructure, tools, and workflows. It has built-in algorithms for linear and logistic regression, among several other statistical software packages.

Amazon SageMaker Canvas is a generic automatic ML solution for classification and regression problems, such as fraud detection, churn analysis, and targeted marketing.

Amazon SageMaker Data Wrangler simplifies the process of data preparation and feature engineering. You can complete each step of the data preparation workflow (including data selection, cleansing, exploration, visualization, and processing at scale) from a single visual interface. SageMaker Data Wrangler provides diagnostic visualization capabilities using lasso. It can plot coefficient values from a lasso model trained on your data. Variables with coefficient values that are close to zero may be redundant and may not contribute significantly to the performance of an ML model.

Get started with lasso regression on AWS by creating a free account today.