What is an NPU?

A Neural Processing Unit (NPU) is a specialized processor for machine learning inference workloads. Although CPUs and GPUs are commonly used in general-purpose machine learning training and inference, NPU offers accelerated processing in AI inference workloads. NPUs offer higher throughput and lower power requirements than CPUs and GPUs, for a good fit for modern AI inference.

Why are NPUs important?

Deep learning, machine learning, predictive analytics, and other AI workloads require immense processing power. Since generative AI has become prominent, companies largely run AI workloads on CPU or graphic processing unit (GPU)-based servers. However, CPUs are not designed to handle low-latency neural processing, while GPUs incur significantly higher energy consumption and costs. An NPU is designed specifically for AI tasks to offer lower energy consumption and lower costs.

Here are some key benefits of NPUs:

NPUs help organizations achieve a lower cost per inference when operationalizing AI models
NPUs allow organizations to shift AI processing closer to the edge, where data is locally stored
NPUs address growing demands for local AI features on computers, mobile devices, and the Internet of Things (IoT)
NPUs offer significantly lower cost per inference, so organizations can reduce cloud spend for AI/ML workloads
NPUs can power cloud data centers or be integrated into a computer.

What are the types of NPUs?

AI chip producers manufacture NPUs exclusively on a standalone chip or integrate them as a System-on-Chip (SoC) with GPUs and CPUs. You can deploy NPUs at different architectural levels to offload heavy AI processing.

Edge NPUs

Edge NPUs are dedicated AI chips that provide local inference capabilities at the network edge. They power remote applications like Internet of Things (IoT), drones, and autonomous vehicles with real-time AI processing.

Cloud NPUs

Cloud NPUs are AI accelerators designed to support large-scale AI processing across data centers. For example, AWS Inferentia is a family of NPUs that deliver cost-efficient, high-performance ML inference at scale.

Embedded NPUs

Embedded NPUs are on-device accelerators built to enhance AI operations on smartphones, laptops, IoT devices, and smart wearables. For example, Apple Neural Engine, Qualcomm Hexagon NPU, and Intel AI Boost are NPUs capable of handling background blurring, photo editing, and noise cancellation.

NPU vs AI accelerator

An AI accelerator, or neural accelerator, is a dedicated AI chip that can serve different purposes within AI. For example, AI accelerators can be dedicated to training, such as the AWS Tranium family of chips. NPUs are a specific type of AI accelerator dedicated to inference, or the response cycle of turning an input from an ML model into an output.

How does an NPU work?

An NPU processes AI tasks in a way that satisfies the requirements of ML neural network inferencing. It consists of specialized computational units capable of processing large amounts of data in parallel. By leveraging a data flow architecture, it can perform inferencing very quickly, solving problems and predicting outcomes at speed. More importantly, NPUs don’t consume as much energy as a CPU or GPU when running matrix multiplication, activation functions, and other data operations during inferencing. Let’s take a smartphone, for example. When integrated with an NPU, you can filter noise in phone calls more effectively using a localized AI model without draining the phone’s battery.

We share key characteristics of an NPU below.

Tensor and matrix operations

NPUs are an AI chip that runs tensor and matrix operations. These can be the basis of neural network operations and are commonly used for facial recognition, predictive modeling, image classification, and other inference tasks. Compared to GPU inference, running these operations on an NPU is more energy efficient.

Dataflow architecture

NPUs support in-memory computing to reduce data movements. Unlike a CPU, which moves data from memory to internal registers for processing, an NPU can process data where it’s stored. This allows low-latency responses to AI queries.

Low-precision arithmetic

NPUs are designed for low-precision mathematical computation. The goal is to deduce an outcome from a multi-matrix of data quickly and learn from the results. We call this quantized inference, where the model computes in INT8 rather than FP32. INT8 means the processor performs integer operations in 8 bits, while FP32 means 32-bit floating-point operations, which result in more accurate results. Despite lower precision, NPUs offer lower physical footprint, energy savings, and low-latency response.

On-chip memory and caching

In AI inference, the time spent moving data from external memory chips to the processor is often greater than the time spent processing. This results in inefficiency, especially in large-scale data processing. NPUs feature large on-chip SDRAM to overcome memory bandwidth bottlenecks.

NPU vs. GPU vs. CPU

CPUs are traditionally used for logical operations and general tasks in computers. They process tasks in sequence, which isn’t optimal for AI processing. Meanwhile, GPUs were originally used to offload video processing tasks from CPUs. GPUs excel in parallel processing, making them suitable for high-performance computing, big data processing, and AI training.

NPUs take a different architecture approach to the parallelism model of GPUs to handle the computational complexity that AI tasks demand. However, NPUs have higher power efficiency than GPUs. NPUs are not meant to replace CPUs and GPUS, but to augment them in AI/ML workflows. For example, organizations use GPUs for training the model, CPUs for running lightweight machine learning tasks, and NPUs for heavy or complex AI predictions.

What are the key capabilities of an NPU?

NPU provides a powerful and efficient solution for AI computation through these capabilities.

Inference acceleration

AI prediction requires a trained model to perform complex operations on unseen data. An NPU performs these operations in parallel, so it can accelerate AI tasks and enable users to get predictions much faster.

On-device AI

Devices running AI workloads often send queries to a cloud server that hosts the ML model. This introduces latency, especially on a congested network, which delays inference results. Deploying an NPU on a device reduces the network round-trip. It allows offline prediction and tighter control over data privacy.

Model compilation and graph optimization

NPUs run AI models that are specifically optimized for performance. ML engineers build AI models using high-level frameworks such as TensorFlow and PyTorch. On their own, these models cannot benefit from NPU’s capabilities. To do so, ML engineers compile the models into a format that the specific NPU understands. For example, you use the AWS Neuron to compile an ML model that runs on AWS Inferentia. Once compiled, the model can infer much faster.

Multi-model and concurrent execution

Some datacenter NPUs can run multiple models simultaneously to support concurrent queries or AI tasks. They use methods like time-slicing and virtualization to distribute AI inference capabilities. For example, you can allocate part of the NPU to image classification and another to video processing.

What are the use cases for NPUs?

Various applications use NPUs to improve inference speed and reduce costs.

Computer vision

NPUs allow computer vision applications to expedite object detection, image classification, and video analytics. For example, smartphones can automatically filter noise from video calls without needing to connect to a cloud-based AI model.

Natural language processing

Many generative AI applications involve speech recognition, machine translation, and text classification. NPUs are specifically designed to handle tensor operations underlying deep learning tasks, such as identifying patterns in text and speech without excessive energy usage.

Recommendation systems

Ad-tech and e-commerce systems use deep learning models to generate real-time recommendations. NPUs can improve customer experience by reducing the time to analyze shopping data and deliver personalized responses.

Robotics and autonomous systems

Industrial applications integrate AI with robotic capabilities to automate critical workflows. These systems often operate in harsh environments with strict power and thermal constraints. NPUs are ideal for handling critical AI processes because of their low power consumption and low heat output.

Generative AI inference

Generative AI applications are usually hosted on a centralized server. Each inference consumes processing power, which is served by GPUs. This architecture, however, can be better optimized for localized applications. By using NPUs, AI providers can reduce the cost per token, reduce response latency, and consistently meet service-level objectives.

What are the key considerations when evaluating NPUs?

NPUs are important for energy-efficient AI deployment. These are criteria to look for when choosing an NPU.

TOPS (Tera Operations Per Second)

TOPS measures the NPU’s processing capability. An NPU with 50 TOPS can process 50 trillion operations per second. However, TOPS shouldn’t be the only metric you use to assess an NPU’s performance. Precision also contributes to the NPU’s inference. You will need to examine the precision, such as INT8, to compare higher TOPS value chips. For example, a 50 TOPS chip with INT8 is not equal to a 50 TOPS chip at INT4. Most NPUs operate with INT8 or INT4 mathematical precision.

Memory bandwidth

An NPU’s performance relies on memory bandwidth. Therefore, consider the memory chip that you’ll use for the AI chip. For maximum performance, high bandwidth memory (HBM) is better, but Low-Power Double Data Rate (LPDDR) memory can prolong battery life for on-device usages.

Software stack maturity

NPUs’ adoption is rapidly gaining traction, but not all NPUs support all use cases, and their toolchains vary by vendor. Ensure that the particular NPU model is supported throughout the AI development cycle. This prevents unnecessary costs that you might incur for fitting the model to the accelerator.

Model portability

You typically cannot deploy the same NPU-compiled model on different accelerators. Different formats are specific to different vendors and are not portable. Plan your hardware requirements to prevent maintaining multiple unnecessary models.

Power envelope

Determine the limits of power the AI model is allowed to operate in, particularly if you’re deploying on-device or edge computing. If you’re using an NPU for a cloud data center, examine its performance-per-watt and choose an AI chip that operates within the limit.

Batch size behavior

Some NPUs are optimized for large batches, while others handle single samples better. Voice recognition requires low-latency inference, while data center AI deployment benefits from high-throughput batch processing.

How can AWS support your NPU and AI inference requirements?

AWS offers a range of purpose-built accelerators and chips for your AI inference applications. Discover AWS NPU instances and machine learning solutions:

Amazon EC2 Inf2 instances Inf2 instances are purpose-built for deep learning (DL) inference. Inf2 instances are powered by AWS Inferentia2, the second-generation AWS Inferentia chip.
Amazon EC2 Trn2 instances, powered by 16 AWS Trainium2 chips, are purpose-built for generative AI and offer high-performance EC2 instances for training and deploying models with hundreds of billions to trillion+ parameters.
Amazon SageMaker is the center for all your data, analytics, and AI. With Sagemaker, you can build, train, and deploy ML models, including FMs, for any use case with fully managed infrastructure, tools, and workflows.
AWS Neuron SDK is the developer stack for running deep learning and generative AI workloads on AWS Trainium and AWS Inferentia.

Get started with AI inference on AWS by creating a free account today.

What Is an NPU?