What Is Edge Inference?

Edge inference is the process of a trained machine learning (ML) model taking an input and generating an output (inference), where that model is located very close to the end-user or data source. Edge inference uses AI models in edge locations to help speed up the time-to-results for users. Edge inferencing is important in time-sensitive applications such as healthcare, industrial operations, and autonomous driving.

There are several different architectural patterns for edge AI inference, with each offering organizations different capabilities and performance constraints. Below are the main edge AI systems.

On-device inference

On-device inference is where processing occurs on an edge device, exactly where the data source lies, for instance, on a self-driving car. There is no dependence on cloud servers with this approach, making its ultra-low latency one of its key differences from other formats. Use cases that need instant data processing may opt for running artificial intelligence on a local device.

Gateway inference

Gateway inference is where an edge device forwards the necessary data to an edge computing node. That node hosts machine learning models and performs AI inference before then transmitting the data on. By moving the processing to this edge location, this mode does increase latency but also allows for more powerful AI processing. Also, organizations can consolidate many inputs from several different local devices, processing them in tandem to save energy.

Fog computing inference

Fog computing inference is a gateway approach in that it has multiple edge nodes and gateways that connect to regional cloud data centers. Fog computing offers edge AI models significantly more processing power while still benefiting from the relatively local data processing. It’s an effective choice for applications that need low latency but also need to deal with very high volumes of aggregated sensor data.

By moving the pre-processing of data and running of machine learning models closer to the end user, this results in reduced latency in AI systems to provide outputs at speed. Edge locations are data centers close to end-users, critical infrastructure sites, or business operations centers.

Several distinct underlying technologies converge for AI inference at the edge.

Model preparation

In order to make edge inference as optimized as possible, organizations first take steps to prepare their model. These include compression, quantization, and pruning to reduce the model size. Inference runtime optimization techniques, such as continuous batching, KV caching, and memory management, allow ML models to minimize compute overhead while maximizing throughput.

By preparing models for optimized inference, you are better able to maintain predictable performance and offer low latency at the edge.

Edge deployment

Deploying ML models at the edge often benefits from container orchestration. By running a model within containers, the orchestrator can automatically scale up or down resource consumption to match demand. The added agility and flexibility that container deployment offers make it a scalable and portable solution that you can move to different sites.

Local processing

Alongside the architectural side of ML models and their deployment, organizations must decide which functions to perform locally and which data to send and process elsewhere.

Edge inference architectures differ by use case. The model is typically able to perform inference locally and send low-priority requests to other processing centers for a hybrid configuration. Low-latency edge inference reduces data transmission overhead, allowing for real-time processing and decision-making. Some edge inference pipelines only transmit summarized features or metadata, rather than the entirety of raw application or sensor data. Minimal data movement reduces bandwidth consumption significantly.

Cloud AI workloads integration

While inference occurs at the edge, cloud integration still offers a range of benefits. For example, the model artifacts can still be hosted in the cloud, helping to provide a centralized structure for training, evaluation, and further model versioning. For new model versions, this visibility also allows for a seamless cross-fleet rollout to edge locations.

If models require multi-model inference or other connections, organizations might also opt for a hybrid model. While data is processed locally, a connection to the central system will provide the extra processing resources needed.

Edge AI inference moves AI processing closer to the data source, enabling rapid and stable inference at the edge. Below are the major benefits of using edge computing infrastructure alongside AI models.

Reduced latency

The central benefit of edge AI is low-latency processing and output. Local data processing at edge locations eliminates the need to transfer volumes of data to distant data centers or a cloud computing facility. Removing this time or significantly reducing it lowers latency, enabling applications that rely on real-time decision-making.

Many of the use cases, such as autonomous vehicles and industrial automation, rely on edge AI’s ability to perform local inference.

Enhanced privacy and security

When you process data on edge devices, you limit its contact with external sources, helping to keep sensitive data private. By minimizing the exposure of data to external networks, organizations can reduce the attack surface of that information, while also more directly meeting regulatory requirements around data transmission and residency.

The risk of data leakage is lower, as AI processing locally gives malicious entities fewer contact opportunities. Highly regulated industries such as medicine or finance can see this as another benefit of edge computing.

Lower bandwidth costs

Edge computing reduces the need for bulky data transmission, with inference at the edge only sending a summary of outputs to centralized data centers. A local data processing approach reduces the need to consume bandwidth while still achieving high performance outcomes.

Improved reliability

Edge AI inference can continue to process data locally and run AI models without major disruptions, even if there is only an intermittent internet connection. Edge devices don’t have to depend on constant cloud connectivity, helping to provide a high degree of stability. For organizations that rely on their edge AI models for real-time data processing in critical situations, this reliability is essential.

Although edge AI inference offers organizations numerous benefits, several challenges need careful consideration.

Hardware constraints

Edge devices and locations may not have the same level of hardware as cloud computing services. If you need to run a large machine learning model, the processing power it requires might exceed device or edge-location limits. Organizations might overcome hardware constraints by focusing on model optimization techniques. Enlisting the strategies featured above can reduce the overall model size and its inference demands, which might allow complex models to run locally.

Model deployment complexity

Edge computing deployment is complex, often requiring numerous platforms, interconnected systems, and runtime environments. Turning to containerization can solve these issues, helping to reduce configuration challenges and create a manageable system that organizations can scale to meet demand.

Security and updates

Cloud servers have numerous layers of security protection that keep them safe. Edge devices, often in locations that are easier to access, don’t have the same comprehensive cybersecurity systems in place to keep them safe. If performing on-device edge inference, organizations need to turn to edge security strategies such as encrypted local storage, device attestation, and regular patching to help keep devices secure.

AWS provides a range of services that help support your edge computing requirements, including AI inference at the edge. Explore our edge services:

• Amazon SageMaker AI helps you build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows at the edge.

• AWS IoT Greengrass is an open-source edge runtime and cloud service for building, deploying, and managing device software. Locally collect, aggregate, filter, and send data.

• AWS Wavelength lets you run applications using AWS Infrastructure and services in AWS telco partners’ data centers to meet your low-latency, data residency, and resiliency needs, including edge inferencing.

Get started with edge inference on AWS by creating a free account today.