What Is Explainable AI?

Explainable artificial intelligence (XAI) is the set of methods and technologies that allow human users to understand and influence AI model behavior and decision-making. One of the main challenges in AI adoption is the need for more trust in AI-generated output, especially in mission-critical business operations. Explainable AI methods describe the internal workings of an AI model, its potential biases, and the reasoning behind how it arrived at a specific output. They answer important questions about the data and steps the AI system uses to arrive at a conclusion. Explainable AI increases transparency, builds trust, and supports the debugging and performance improvement of AI models.

Advanced AI models are deep neural networks with multiple hidden layers. Understanding how they behave is challenging, even for machine learning engineers. Organizations hope to interpret and explain how deep learning models work for various reasons.

Build trust

Machine learning models are similar to black boxes, whose inner workings are hidden from plain sight. Adopting explainability allows machine learning engineers to confidently explain how a specific AI system operates. With explainability, the intended users are more likely to understand and trust the results that AI predicted.

Improve productivity

Machine learning engineers continuously try to improve model performance even after AI services are deployed. Explainable AI provides frameworks and tools to visually investigate model behavior and optimize performance. This way, organizations can more easily address bias, inaccuracies, or other anomalies.

Accelerate business innovations

AI technologies promise to transform businesses, but only if they are precise, consistent, and trustworthy. Organizations can set up an AI governance committee to ensure that the deep learning technologies they implement fulfill such requirements. With an explainable AI model, they can investigate and analyze how AI can target and significantly impact specific business processes.

Comply with regulations

Some organizations are compelled by law to safeguard users' interests and mitigate risks when using AI in mission-critical applications. For example, healthcare and financial solution providers must demonstrate explainability in the AI systems they deploy to comply with regulatory acts.

Certain key principles help organizations increase explainability in the AI technologies they implement.

Fairness and debiasing

AI algorithms may produce biased and unfair outcomes toward specific groups or demographics. To mitigate such outcomes, engineers apply debiasing techniques to detect factors leading to such patterns and refine the AI model.

Human centrism

Rather than creating AI systems to replace humans, explainable AI principles seek to augment human capabilities with artificial intelligence. This makes the workforce more productive and valued, contributing to the organization's growth.

Privacy and security

Engineers must ensure adequate data security and privacy protection to make AI models more explainable. When explaining how a machine learning model reaches its prediction, sensitive training data should be protected from unauthorized access.

Transparency and audibility

Explainable AI forms part of the components for responsible AI, an approach that helps organizations adopt AI safely and enable model governance. Internal and external auditors should be able to assess AI models to ensure they meet legal and compliance requirements.

An explanation in AI can be thought of as the answer to a Why question that helps humans understand the cause of a deep learning model's prediction. It might answer questions such as:

• Why did the model predict a negative outcome, such as a loan rejection for a given applicant?

• How does the model make predictions?

• Why did the model make an incorrect prediction?

• Which features have the largest influence on the behavior of the model?

Scientists classify explanations into different categories to better explain AI models.

Global explainability

Global explainability provides a broad understanding of how an AI model works. Specifically, it uncovers key features that greatly influence the model's output. For example, an AI-based credit-scoring system considers age a significant factor when approving loans.

Cohort explainability

Cohort explainability allows engineers to investigate why the AI model behaves in a particular manner for a group of training data. For example, the AI credit scoring system rejects most applications from people in their 20s. This approach helps ensure model accuracy during testing and addresses ethical concerns.

Local explainability

Local explainability is helpful for monitoring whether a model's performance accuracy has gradually declined over time. It investigates individual decisions that a model makes during actual usage. For example, the AI system rejects a loan application, which it should normally provide. This prompts engineers to investigate possible factors influencing the model's decision.

Explainable AI techniques can be grouped into two broad categories.

Explainable machine learning (ML)

These methods attempt to explain a model's decisions after they are made. You can think of them as post hoc explanations of black-box models. Some issues that can occur with these methods are a lack of robustness, such as being easily fooled by adversarial attacks and ambiguity, where multiple "explanations" may exist. Machine learning engineers use different techniques to evaluate model explability. We share popular ones below.

LIME

Local interpretable model-agnostic explanations (LIME) try to explain a part of the model's behavior. LIME feeds varying data points to a more interpretable surrogate model. The surrogate model then generates outputs, which the original model evaluates. The goal is to fine-tune the surrogate model to closely resemble the original model's behavior. Because the surrogate model is interpretable, scientists can understand critical features that influence the model's results.

SHAP

Shapley additive explanation (SHAP) helps engineers understand the contribution of individual features to a model's output. SHAP is based on game theory, where different features can contribute individually and as a whole. When used to interpret models, SHAP compares the model output to a baseline determined by various input features.

PDP

Partial dependence plots (PDP) visualize the relationship between input features and the model outcome. PDP can examine whether the model output is linear or non-linear when measured against the input feature. Engineers also use PDP to observe how two features interact in a complex model.

DeepLIFT

Deep learning important features (DeepLIFT) is a method that determines the importance of different input features by backpropagating from the output. It compares neuro predictions of real-world data against a reference baseline and scores them accordingly. DeepLIFT is fast and can reveal feature relationships that other methods overlook.

Interpretable machine learning (ML)

These methods take advantage of the fact that specific AI models are, in some sense, interpretable in the way they work. They can check and understand the inner workings of the model.

Expressive boolean formulas

Expressive Boolean formulas are a series of classifiers that perform logical operations on a set of variables. Depending on the input variables, each classifier's output can be positive or negative. Together, they act as an interpretable model that helps engineers understand the complex decision-making algorithms applied in black-box AI systems.

Integrated gradients

Integrated gradients identify important features that impact how a model classifies an object. You can think of the gradient as the learning curve a model goes through based on the output feedback. The technique compares model input and a baseline, which involves interpolating the latter to resemble the desired image. Then, it identifies features that strongly affect model prediction by calculating the gradient changes throughout the transformation.

Explainable boosting machine

The explainable boosting machine (EBM) is inspired by additive modeling, such as linear regression. It can interpret complex model responses by representing input features with multiple smaller decision trees. Each tree focuses on predicting the outcome associated with one feature. EBM then summarizes all decision tree responses to interpret the model behavior.

Measuring bias in ML models is the main way to evaluate models for explainable AI.

Pretraining bias

Pretraining bias happens when training data that the AI model learns from doesn't fairly represent real-world distribution. This occurs when training datasets are not properly curated or there is limited real-world data to work with. For example, healthcare data might be biased because of insufficient data from specific age groups. Machine learning engineers detect pretraining bias by evaluating training data against bias metrics.

Post-training bias

Post-training bias refers to inaccuracies or unfair predictions resulting from biased training data or machine learning processes. When assessing post-training bias, engineers compare model predictions to data labeled by humans. They also use ML bias analysis tools to measure model fairness.

Amazon SageMaker is a fully managed service that combines a broad set of tools to enable high-performance, low-cost deep learning for any use case. Amazon SageMaker Clarify provides tools to help explain how deep learning models make predictions. These tools help stakeholders understand model characteristics as a whole before deployment and debug predictions supplied by the model after it is deployed. For example, Amazon SageMaker Clarify supports

• Foundation model (FM) evaluation will help you quickly evaluate, compare, and select the best FM for your use case based on various criteria across different tasks within minutes.

• Model explainability for tabular, computer vision, and time-series models during model development or post-model deployment.

You can use the bias and explainability reports to identify potential issues. Direct efforts to improve accuracy, remove bias, and increase performance. Build trust in your AI models and support compliance.

Get started with explainable AI on AWS by creating a free account today.