What is a Support Vector Machine?

A support vector machine (SVM) is a machine learning algorithm that classifies data by analyzing and filtering data points in an n-dimensional space. All machine learning (ML) models analyze training data and attempt to find relationships between them. They then use these relationships to predict outputs for new, unknown inputs. A support vector machine plots data points in an n-dimensional space and then attempts to find a hyperplane or data space that separates the points into different data classes. It requires training with labeled data for the appropriate classification. It can also be used for regression analysis to predict exact output values (instead of categories) for unknown input values.

What are the use cases of support vector machines?

A support vector machine algorithm is primarily used for binary data classification. Binary data classification means determining whether a data point is from group A or group B. Other use cases include:

Facial detection
Handwriting recognition
Speech recognition
Image tampering determination
Medical classification
Recruitment testing
Weather reporting.

Example application

An example of where an SVM algorithm would be useful in binary data classification is when determining whether a given dose of medicine is effective. Too small a dose and the issue or condition remains. Too high a dose of medicine and the side effects outweigh the benefits. Here, the dosage amounts are one-dimensional input data (e.g. 100mg, 250mg, 400mg). However, to divide the data into two classes or groups(effective and not effective), we cannot use a linear classifier on the data. Instead, we can map the data to an x and y axes and transform it into a two-dimensional space via a feature mapping function or kernel function.

Analysis of historical data reveals that the value (x), when squared to obtain a corresponding y value, is a good indicator of effectiveness. Plotting the new values on a 2D space can show where a linearly separable line can be drawn to distinguish the two classes: an effective dose and a non-effective dose.

How do support vector machines work?

When data points don’t fit a simple linear logistic regression algorithm, there may be another way to slice the data into two parts so that it has a better fit and, thus, better output estimates. An SVM algorithm works by transforming data into higher-dimensional space to help find a slicing method to attain that better fit. SVMs come in a range of different types, but this basic premise is the same, no matter the algorithm behind it. SVMs are more grounded in mathematics than computer science.

Hyperplane

A hyperplane is a straight-line-based plane that separates a support vector machine’s data points into two classifications. When data is plotted onto a space with two axes—x and y—a hyperplane separates two different data classifications, one above and one below, with a simple straight line.

When data is plotted into three-dimensional space, the hyperplane for linearly separable data is a flat plane, like a piece of paper, with data classified by falling above or below this hyperplane. In n number of dimensions, it is another hyperplane, with linear dimensions, to split the space into two different regions.

Decision surface

In other types of nonlinear data SVMs, the plane itself in the higher dimensional feature space may not be a linear decision boundary. In these cases, it is called a decision surface. The optimal hyperplane is the furthest possible plane between the two different categories of data sets and is also known as the maximum margin hyperplane.

Support vectors

The data points that are closest to the hyperplane or decision surface are the support vectors (SVs). These support vectors are the points that are most influential on the placement of the plane, and thus also when conducting new data classifications with unknown data. The SVs closest to the plane fall on the margins.

A maximum margin hyperplane is designed to have the widest margins between SVs on either side of the plane.

Different kernel functions

A kernel function allows data projections into higher dimensions to determine a hyperplane so that data that doesn’t sufficiently fit a particular mathematical function in a lower dimension can be more accurately classified. It then measures the similarity between two data points in the higher dimension.

Kernel trick

Kernel functions transform regular data into “n” dimensions implicitly. Once a hyperplane is established, they calculate the distance from the new point to the hyperplane. Instead of transforming the data, a mathematical equation calculates the similarity rather than modeling the new-dimensional data first. Using a kernel trick function, there is no need to do the mapping and conversion.

What are the types of support vector machines?

There are multiple different types of kernel functions, resulting in different types of SVMs. Instead of hyperplanes, some of these kernel functions have non-linear-based slicing of spaces, where decision surfaces are used instead.

Soft-margin SVM

It’s not always possible to find a hyperplane that neatly separates the data into two distinct, linearly separable classifications, even in n dimensions. In this case, we want to use a soft-margin SVM. Soft-margin SVMs are less sensitive to outliers and thus tolerate miscalculations. The figure representing this in calculations is the regularization parameter (C).

A soft-margin SVM differs from a hard-margin SVM in that existing data points may lie within the margins designated by the support vector data points surrounding the hyperplane.

Other techniques to deal with outliers include smoothing algorithms to remove outliers, such as z-score, winsorization, and other statistical methods.

Linear kernel

A linear kernel, or a linear SVM, maps data so that it can be classified as falling either above or below a single line; the data is linearly separable. A linear kernel function is a simple dot product, where K(x,y) = x^t.y

Polynomial kernel

Up until now, SVMs have been described as classification algorithms that divide space into classification areas using straight-line hyperplanes. A polynomial kernel instead maps data to a feature space, with the boundary being a polynomial-based decision surface. While linear SVMs are fairly simple in their mathematical complexity, non-linear SVMs are more complex, albeit less so than models like convolutional neural networks.

The corresponding kernel function, or SVM algorithm, is K(x,y) = (x^t.y+c)^d

Other types of kernels

The Gaussian radial basis function kernel maps data to an infinite number of dimensions to find the best-fit decision surface in infinite dimensions. The sigmoid function kernel maps data to a hyperbolic tangent. Other lesser-used kernels include the Laplacian kernel, the Rational Quadratic kernel, and the ANOVA kernel.

Support vector regression is a particular type of SVM that is used to calculate continuous values with regression tasks rather than for classification.

How can AWS help?

Amazon SageMaker is a fully managed service for preparing data and building, training, and deploying ML models for any artificial intelligence use case with fully managed infrastructure, tools, and workflows. It provides several built-in general-purpose algorithms that can be used for either classification or regression problems. The Amazon SageMaker Linear Learner algorithm solves classification and regression problems. It includes eight functions, one of which is an SVM for binary classification problems.

For organizations looking for out-of-box solutions, Amazon Comprehend is a fully managed service that uses ML to uncover valuable insights and connections in text. The Custom Classification API lets you easily build custom text classification models using your business-specific labels without having to implement SVM from scratch.

Similarly, Amazon Macie is a data security service with an SVM classifier that discovers sensitive data using ML and pattern matching. Macie uses an SVM classifier to classify the content within S3 objects in addition to the metadata of your files. Macie trains the SVM classifier using data of varying content types optimized to support accurate detection of data content, even including the source code you may write.

Get started with support vector machines on AWS by creating a free account today!

What Is a Support Vector Machine?