Architecting multi-carrier interoperability with Edge Discovery APIs on AWS Wavelength

This blog was written in collaboration with OpenSesame and the 5GFF Technical teams from Vodafone, Rogers and Verizon.

On March 1, 2023, at Mobile World Congress (MWC) in Barcelona, the 5G Future Forum (5GFF) and Open Sesame Media, Inc. (Open Sesame) demonstrated a first-of-a-kind cross-operator music jam session using AWS multi-access edge compute (MEC) technologies like AWS Wavelength and AWS Outposts. In this demonstration, 3 world-renowned guitarists performed a medley of rock ‘n roll selections – each sitting in separate countries – through the low-latency compute environment using 5GFF’s Edge Discovery Service (EDS) API, with Open Sesame’s SyncStage ultra-low latency audio pipeline, with the 5G connectivity of Verizon, Rogers and Vodafone and AWS MEC.

This demo not only achieved multi-country, glass-to-glass latency that enables the guitarists to jam together online, but also unlocked a new reference pattern for multi-edge, multi-region applications using an interoperable EDS API. In this blog post, we introduce the fundamentals of this architecture and present best practices for how to operate a MEC application at-scale.

The AWS MEC vision consists of a Cloud continuum. Our customers want to deliver applications to end users and devices with ultra-low latency or high bandwidth for data-intensive workloads. It is important to provide the same experience while building on public cloud and the edge. AWS has built a continuum of consistent cloud services to support businesses and their customers. This includes public cloud in region, public MEC (AWS Wavelength), private MEC (AWS Outposts) in our customers data centers, and finally at the edge of the network to support Internet of Things (IoT) and rugged use cases. This allows applications to be deployed across the entire continuum, based on their requirements.

Go Global Edge in Minutes with AWS Wavelength

Along with bringing compute and storage closer to 5G-connected devices, AWS Wavelength offers customers lower latency, higher bandwidth, and a stronger security posture. A secondary benefit is the global reach of the service itself; with partners like Verizon, Vodafone, and more. Developers can now go global in minutes by building edge applications that span 1 or more MEC nodes (Wavelength Zones, or Outposts), not just within a single region, but across multiple AWS regions – such as the Boston and New York city Wavelength Zones in the us-east-1 (N. Virginia) Region. Instead, applications can extend across multiple Wavelength Zones, each within separate regions, such as the Manchester Wavelength Zone in the eu-west-2 (London) Region alongside the Miami Wavelength Zone is the us-east-1 (N. Virginia) Region. Application developers can therefore create a more densified, geo-distributed footprint of compute resources, allowing mobile clients to experience a more uniform, low-latency experience.

While AWS Wavelength Zones (WLZ) have been used in hub-and-spoke architectures, such as database replicas syncing data back to a primary running in the parent AWS Region, few have taken advantage of the capabilities of data exchange across multiple Wavelength Zones. Traditionally, within a given AWS Region, spoke-to-spoke communication is only supported over Carrier IP addresses. Hence developers rarely experimented with applications reliant on cross-WLZ communication. With this global audio synchronization use case, cross-WLZ communication becomes an architectural requirement.

For real-time applications such as video conferencing, live streaming or gaming, users’ quality of experience (QoE) is dependent on keeping the audio and video streams in sync with minimal delay. Real-time applications rely on end users’ access medium (e.g., cellular, home broadband) to deliver an optimal experience and are further challenged when application users span hundreds, if not thousands, of miles from one another. Long distance data transmission will incur a certain amount of latency, so even the fastest possible network connections cannot eliminate it. Designing audio sync applications to manage this delay, while effectively synchronizing the audio streams, is crucial to make these use cases viable.

Minimizing audio latency between a group of musicians is critical for musicians to play, hear, and react to each other’s performances. There is almost no latency when musicians perform in the same physical location. Those who have attempted to remotely perform online with common digital collaboration solutions have experienced challenges with synchronously performing together due to the significant audio latency.

About Open Sesame Media’s SyncStage

Open Sesame Media, Inc. is a B2B venture providing low latency synchronized audio to application developers via its patented SyncStage^TM digital audio platform. SyncStage allows application developers to create synchronous low audio latency experiences within their applications to enable their users to collaborate with others in ‘real time’. Combining SyncStage with 5GFF’s Edge Discovery Service (EDS) APIs enables ultra-low latency synchronous audio collaboration and experiences for musicians to perform together while being hundreds of miles apart.

Julian McCrea, CEO of Open Sesame stated, “For our specific implementation, we were using the Edge Discovery API to work with musicians in each of the 3 countries, using SyncStage. That ability to be frictionless for the end musician – for them to be able to open their phone, find another musician, and start jamming and performing (with the Edge Discovery API helping them find the correct Edge location in each territory) – makes collaboration borderless with SyncStage.”
The Open Sesame SyncStage platform consists of three major components:

• SyncStage Backend: Built on Amazon API Gateway, the Backend enables session control across all the participating devices during a session. At the MWC demo, all the devices connected to the Backend were deployed in AWS us-east-1 Region. The Backend checks with the 5GFF EDS APIs and provides optimal MEC service endpoint address and connection information to the devices in real time.
• Studio Server: Deployed on Amazon EC2 instances, the Studio Server streams content to and from the connected devices. An instance of Studio Server is deployed at each MEC location, and the devices connect to their nearest instance via 5G network.
• SyncStage SDK: A software only SDK for application developers to integrate into their application that provides ultra-low audio latency to a group of users to enable a vast range of synchronized audio experiences.
• Mobile Application using SyncStage SDK: During the demo, the musicians plugged in their instruments and monitor to a 5G connected smartphone, running the application.

In New York and London, Open Sesame deployed the Studio Server instances at the local Wavelength Zones. In Toronto, the Studio Server was deployed on AWS Outposts in a Rogers Data center, connected to the Rogers 5G network to resemble the setup in the other locations. Each of these instances were launched in a private subnet within a VPC spanning across the MEC and their parent Region.

Once Open Sesame deployed the applications, and connected devices to their local 5G networks, how do we interconnect them all? This was accomplished using a networking concept in AWS known as VPC peering. A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different Regions (also known as an inter-Region VPC peering connection). VPC peering allowed these disparate instances in private subnets to route data to each other using the AWS global backbone.

The following diagram summarizes the architecture used during the demo.

1. Guitarist A in the geographic proximity of the New York metropolitan area records audio that is captured by a mobile device connected to the Verizon Wireless 5G network. That mobile device communicates with the Verizon Edge Discovery Service API (via SyncStage Backend) to retrieve the IP address and connect to the most optimal Wavelength Zone.
2. Once the EDS API responds with the optimal Wavelength Zone (NYC) and the specific endpoint Carrier IP address (eg, 155.146.X.Y), the device streams its audio over UDP through the radio access network (RAN) to the New York Wavelength Zone. The Studio Server in the NYC Wavelength Zone forwards the traffic to the Amazon API Gateway in the region so it can be proxied to the two other guitarists, each in separate AWS Regions. One traffic flow is sent over the VPC Peering connection between us-east-1 (N. Virginia) and ca-central-1 (Canada) and a separate traffic flow is sent over a second VPC Peering connection between us-east-1 (N. Virginia) and eu-west-2 (London).
3. In the first traffic flow, the traffic is forwarded directly to the Private IP address of the EC2 instance in the AWS Outposts Rack operated by Rogers in the Toronto metropolitan area. Once the traffic traverses over the AWS global backbone to the ca-central-1 region, it moves over the service link configured between the region and the AWS Outposts. This service link could either be a Direct Connect (DX) connection, a Site-to-Site VPN (S2S VPN) connection or a connection over the public internet. Once in the Studio Server at the Toronto edge facility, the audio stream is sent to Guitarist B jamming in the Toronto metropolitan region.
4. In the second traffic flow, the traffic is forwarded directly to the Private IP address of the EC2 instance in the London AWS Wavelength Zone operated by Vodafone in the London metropolitan area. Once the traffic traverses over the AWS global backbone to the eu-west-2 region, it moves over the service link: a highly available, high bandwidth connection between the AWS Region and the Wavelength Zone maintained by the customer. Once in the Studio Server at the London edge facility, the audio stream is sent to Guitarist C jamming in the London metropolitan region.
5. Around the same time, Guitarist B in the geographic proximity of the Toronto metropolitan area also records audio that is captured by a mobile device. That mobile device communicates with the Rogers Edge Discovery Service API (via SyncStage Backend) to retrieve the IP address and connect to the most optimal MEC location.
6. Once the EDS API responds with the optimal MEC location (Outposts in Toronto) and the specific endpoint Customer-owned IP address, the device streams its audio over UDP through the radio access network (RAN) to the Toronto MEC node. It can now not only produce an audio stream to Guitarists A and C but receive the in-sync audio stream from Guitarists A and C.
7. Guitarist C in the geographic proximity of the London metropolitan area also records audio that is captured by a mobile device. That mobile device communicates with the Vodafone Edge Discovery Service API (via SyncStage Backend) to retrieve the IP address (141.195.X.Y) and connect to the most optimal Wavelength Zone.
8. Once the EDS API responds with the optimal Wavelength Zone (London) and the specific endpoint Carrier IP address (eg, 123.10.X.Y), the device streams its audio over [protocol] through the radio access network (RAN) to the London Wavelength Zone. It can now not only produce an audio stream to Guitarists A and B but receive the in-sync audio stream from Guitarists A and B.

While testing the demo setup, we observed musician-to-musician network latency. Some of this latency is attributed to the physical distance that the audio needs to travel to reach each performer. We experienced lower latency values between Toronto and New York, attributed to the shorter physical distance between these two cities. The longer distances between North America and the United Kingdom incurred greater latency. However, these latencies were neither noticeable to the live performers, nor was it evident in the session recording.

This is the first time that the 5GFF, Open Sesame, and AWS have demonstrated the ability to connect MEC nodes with each other, across multiple geographies – a possibility that did not even exist a few years ago. When a device connects to the nearest MEC node, the traffic is securely sent to the other participants with minimal jitter, compared to routing traffic over the public internet.

Best Practices for Multi-Region Edge Deployment

This reference pattern highlights several critical best practices to design highly distributed, multi-region edge applications. While geo-distributed applications across multiple Wavelength Zones in a single region have been discussed in a prior post, different design considerations are needed when invoking multiple Wavelength Zones across separate regions.

First, to ensure the Wavelength Zones can intercommunicate, peer each of the VPCs to one another, as Transit Gateways do not support Wavelength Zone attachments. Another approach could be to configure a Transit VPC using a software VPN and create a separate VPC in one of the AWS Regions, such as us-east-1.

Second, keep in mind that only Wavelength Zones across separate AWS Regions can intercommunicate directly over Private IP addresses. If two Wavelength Zones within the same AWS Region seek to communicate, they must use their Carrier IP address.

Third, as the number of AWS Regions involved in the application increases, an infrastructure-as-code service such as AWS CloudFormation is critical for reusability.

In partnership with AWS MEC offerings, 5GFF partners offer you the capability to deploy applications and services at the edge of the 5G network. This infrastructure is specifically designed to meet the needs of applications that require low latency and/or high bandwidth, including gaming (including AR/VR), media processing, e-commerce, social media, medical image analysis, and machine learning inferencing apps. By deploying the latency sensitive components of your application at the MEC, you can start taking advantage of EDS today.

About 5G Future Forum

The 5G Future Forum (5GFF) was founded in 2020 to accelerate the deployment and adoption of 5G and MEC applications by fostering interoperability between telecom operators. The 5GFF is working to develop API specifications for MEC-related APIs that will make multi-operator deployment easier for developers. Additionally, the 5GFF has launched the 5G MEC Acceleration Program (5G MAP) for developers to gather feedback on APIs in development in return for opportunities for demos and direct access to some of the world’s largest telecom operators. The full list of APIs and a link to the 5G MAP can be found at 5gff.org.

Learn More

To learn more about this demonstration, you can watch the video of the 5G Future Forum MWC demo, 5GFF & Open Sesame White Paper, contact the team at Open Sesame, browse the AWS Wavelength developer guide, or visit the EDS API documentation from the 5G Future Forum.