What Is Naïve Bayes?

Naïve Bayes is a machine learning algorithm you can use for classification tasks after training it with labeled data. Classification algorithms typically predict the class of an element by determining the relationship between feature values and labels in the training data set. For example, to classify an email as spam or not, the classification algorithm may consider the frequency of words like offer, discount, and click alongside other features like subject headers.

Naïve Bayes assumes that the features are independent given the class label. This means that the presence or absence of a particular feature does not affect the presence or absence of any other feature, simplifying the computation. Despite its simplicity and the unrealistic independence assumption, Naïve Bayes performs well in many practical applications, especially for text classification problems.

Bayesian statistics is a mathematical study that applies conditional probability to solve statistical problems. Conditional probability is an analysis of the likelihood of a particular event occurring based on another event.

We can express the conditional probability formula in Bayesian statistics as:

P(A ∣ B) = P(B ∣ A) x P(A) / P(B)

Where:

• P(A ∣ B) is the probability of event A happening, given event B has occurred.

• P(B ∣ A) is the probability of event B happening, given event A has occurred.

• P(A) and P(B) are the independent probabilities of events A and B.

To understand Bayesian statistics, let's consider the conditional probability of getting an odd number smaller than 3 when rolling a fair die. In this case,

• Event A is the act of rolling 1, 3 and 5.

• Event B is the act of rolling 1 and 2.

Hence,

P (A) = 3/6 = 1/2.
P (B) = 4/6 = 2/3.
P (B ∣ A) = 1/3.

So, the probability of getting an odd number when the die shows a number smaller than 3 is.

P(A ∣ B) = ( 1/3 x 1/2 ) / 1/3.
= 1/6 / 1/3.
= 1/2.

There is a 50% probability of rolling an odd number, which is 1, from a number smaller than 3.

Bayes theorem expands on conditional probability expressed in Bayesian statistics by predicting a specific outcome based on various predictors. It allows data scientists to examine multiple causes affecting an event by expanding the primary Bayesian expression to:

P(Ai ∣ B) = P(B ∣ Ai)P(Ai) / ( P(B ∣ A1)P(A1) + (P(B ∣ A2)P(A2) + ⋯ + P(B ∣ An)P(An) )

Unlike the basic conditional probability formula, Bayes theorem allows subsequent computation of the future outcome as more evidence becomes available. For example, consider the probability of drawing a black ball from a box randomly chosen from the following:

• Box 1: 10 black balls, 5 white balls.

• Box 2: 10 black balls, 10 white balls.

• Box 3: 10 black balls, 15 white balls.

Then, we define B as the event of drawing a black ball.

We can define the initial probability of picking a specific box as

P (A1) = P(A2) = P(A3) = 1/3.

Then, we can express the probability of drawing a black ball from respective boxes as follows.

P(A1/B ) = 10/15
P(A2/B ) = 10/20
P(A3/B ) = 10/25

With that, we can calculate the likelihood a specific box was chosen if we drew a black ball.

Naïve Bayes repurposes the Bayes theorem to enable supervised machine learning for classification tasks. Like the Bayes theorem, the Naïve Bayes algorithm predicts the outcomes of events through conditional probabilities. However, Naïve Bayes slightly alters how the Bayes theorem is applied.

Assumptions

Naïve Bayes classifiers differ from discriminative classifiers like logistic regression, which evaluates which feature is more important when classifying data. Instead, the Naïve Bayes classifier assumes that:

• Every feature or predictor is independent.

• All predictors carry equal weight in determining the final outcome.

• No single feature is more important than others when making predictions with Naïve Bayes. Changes in one feature do not affect the probabilities of other predictors.

Modification

Considering the new assumptions, we can represent the Bayes theorem, P(A ∣ B) = P(B ∣ A) x P(A) / P(B), with conventions that fit the Naïve Bayes principles below:

Posterior probability = Conditional Probability x Prior Probability / Evidence

By doing so, we get the following definition.

• P(A ∣ B) is the posterior probability.

• P(B ∣ A) is the class conditional probability.

• P(A) is the class prior probability.

• P(B) is the evidence or predictor of prior probability.

Example

We want to calculate the likelihood of successful loan approvals based on the applicant's age, occupation, and income. By applying Naïve Bayes, data scientists approximate the probability of loan approval based on the respective predictors. The algorithm considers each classifying variable equally and separately, dismissing any possibility that one could affect the other.

With such modifications, Naïve Bayes becomes a simple and efficient machine learning algorithm for classification tasks. That said, it's important to note that Naïve Bayes' apparently oversimplified assumptions don't accurately represent how predictors behave in real life. For example, most applicants who earn a higher income are likely to have spent more years working in an industry. Still, despite oversimplification, the Naïve Bayes model is highly accurate, making it ideal for sentiment analysis, spam filtering, and text classification applications.

To better understand the Naïve Bayes classifier and its components, imagine a spam filter trained with 10 labeled emails. Each email is labeled based on the occurrences of specific words, the sender's IP address, and the domain name. The spam filter applies a Naïve Bayes classifier to predict the conditional distribution of each feature given the class variable of spam and non-spam.

Posterior probability

The posterior probability is a prediction that updates itself after considering new information. In the above example, the posterior probability is the likelihood of an email being classified as spam because it contains specific words like discount.

Class conditional probability

Class conditional probability is the likelihood of various features or predictors affecting the outcome. It divides the feature class with the target class. Feature classes are categories determined by predictors like words, IP addresses, and domain names. Meanwhile, the target class is the categorical outcome of spam or non-spam. Let's say out of the 4 spam emails, 3 consist of the word discount. So, the class conditional probability for the word discount in spam emails is 3/4.

Class prior probability

Class prior probability computes the likelihood for a specific feature to occur from the entire dataset. For example, 7 emails, including spam and regular emails, contain the word discount. Therefore, the class prior probability for the word discount is 7/10.

Evidence

Evidence, or class predictor probability, calculates the chances of deducing a target outcome based on the available data samples. Let's say the dataset contains 4 spam emails and 6 regular emails. So, the class predictor probability of spam is 4/10.

Naïve Bayes isn't limited to a single machine learning algorithm but a broad group of different models sharing the same conditional probabilistic principles. We share three types of Naïve Bayes classifiers below.

Gaussian Naïve Bayes classifier

The Gaussian Naïve probabilistic classifier assumes a normal distribution for all features. Each feature, or predictor, is distributed over a symmetrical bell curve, peaking in the median and gradually decreasing as they approach the lower and upper tails. Data scientists use this model to classify continuous values by sampling them across the Gaussian curve.

Multinomial Naïve Bayes classifier

The Multinomial Naïve Bayes model makes predictions by sampling features from a multinomial distribution. A multinomial distribution is a dataset that consists of limited or discrete outcomes. For example, the possibility of rolling a specific number with a dice is multinomial. Data scientists often use this type of Naïve Bayes for documentation classification.

Bernoulli Naïve Bayes classifier

The Bernoulli Naïve Bayes classifier works similarly to the Multinomial Naïve Bayes classifier. However, Bernoulli Naïve Bayes classifiers use boolean predictors instead of discrete data points and their frequencies. Boolean is a logical representation of information as true or false. You can use Bernoulli Naïve Bayes classifiers to determine whether customer feedback is positive or negative.

Naïve Bayes is easier to train and implement compared to other classification models. Data scientists can easily train Naïve Bayes learners even with limited training samples.

By assuming conditional independence, Naïve Bayes ignores influencing factors by other features in the dataset. This means Naïve Bayes requires less compute power than discriminative models like logistic regression.

Naïve Bayes excels at classifying datasets with a large number of features. It also consistently and accurately performs multi-classification tasks with continuous data.

While Naïve Bayes is a versatile classification algorithm, there are limitations that you should be mindful of.

• Naïve Bayes decouples all class conditional feature distributions for speed and simplicity. It assumes that all predictors are independent, which is seldom true in real-life applications.

• Naïve Bayes is susceptible to zero frequency, a condition where a particular class predictor does not exist in the training dataset. Zero frequency can result in inaccurate predictions unless smoothing techniques are applied to the dataset.

Amazon SageMaker is a fully managed service that brings together a broad set of tools to enable high-performance, low-cost machine learning (ML) for any use case. The Amazon SageMaker linear learner algorithm provides a solution for both classification and regression problems. The linear learner algorithm provides an increase in speed over the Naïve Bayes technique. With the SageMaker algorithm, you can simultaneously explore different training objectives and choose the best solution from a validation set.

For customers seeking a more readymade solution, Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find meaning and insights in text for customers seeking a more readymade solution. You can extract key phrases, understand sentiment, or identify the main topics in a text document. Amazon Comprehend provides document processing APIs to help you process documents with ease and at scale without implementing Naïve Bayes for text classification.

Get started with Naïve Bayes classifier use cases on AWS by creating a free account today.