XR Streaming: How Holo-Light Solves Major Problems of Augmented and Virtual Reality

Guest post by Philipp Landgraf, Head of Technology, Holo-Light

Augmented reality (AR) and virtual reality (VR) are emerging technologies opening up real opportunities for innovation to enterprises and consumers in all kinds of industries. But today´s XR (extended reality, a term for all immersive technologies) environment also struggles in realizing its full potential. One of the major issues many XR developers and end users face, is the limited computing and graphics power of mobile devices. As performance is limited, the visualization of 3D content in AR/VR is also restricted. The quality and richness of effects are far below what we are used to from modern pc programs. An issue that won´t go away, as there are physical barriers as well as the need to reduce the form factor to mainstream the technology.

Remote Rendering and Streaming of XR Applications

This is where Holo-Light comes into play. Since 2015, we focus on immersive software and technologies. In augmented and virtual reality, we see a big driver for global digitization and a new way of experiencing and interacting with content – from the industrial sector to entertainment and gaming.

With this mindset we have developed a cross-platform remote rendering SDK, ISAR – Interactive Streaming for Augmented Reality, that allows users to securely stream entire big data XR applications to their mobile devices using on-premise resources or, even better, from the cloud. ISAR is easy to integrate into third party apps and already empowers our own out-of-the-box AR Engineering Software ARES Pro for the industrial sector, where the work with data-intense 3D CAD models requires powerful performance and extra privacy.

How the ISAR SDK Works

Once the ISAR SDK is built into your XR application, the app no longer needs to be installed directly on an end device. Instead, the XR app (with the server-side SDK component) is installed on a more powerful server in the AWS cloud. The user simply connects from a by Holo-Light provided gateway (client app) on the mobile device to the XR app on the external server. Now, the rendering process shifts from the low-performance XR device to the high-performance server.

The client app sends data – sensor data for room tracking, position data and hand gestures – to the server with the actual XR application. The data is processed on the server and the 3D content to be displayed is sent back to the client app. In order to guarantee a high-quality experience for the user, this process has to happen in real-time.

How AWS Helped Us Reach the Next Level

Talking about high-quality experience, I have to mention our collaboration with the AWS Prototyping Team. They gave us the kick-start to take our ISAR solution to the next level. At the beginning, honestly, we had little knowledge of the AWS ecosystem. Nonetheless, we always felt the way to go into the future is with the right cloud infrastructure. We talked to AWS about what we envisioned, and right away the AWS Prototyping Team was very helpful in guiding us to the right services and explaining how they would work and interact.

Together, we developed a first working prototype within a few weeks. A scalable architecture our development team could build upon. And that brings me back to what was so great about this collaboration. The AWS Prototyping Team didn’t just help us build the architecture, they managed to transfer their knowledge in a way that empowered us to continue developing on our own.

So, what exactly did we do with AWS? Before, every user needed an expensive high-performance computer acting as a server in order to make use of ISAR. Also, some setup time for the application was needed. The AWS infrastructure now enables the remote rendering of any ISAR empowered app out-of-the-box without any setup. We integrated the ISAR SDK in our own software ARES Pro. This allowed us to create a platform with user management as well as the opportunity to store and upload files. Now users only need a XR device with the ARES Pro client app installed and login credentials to our platform.

The AWS / ARES Pro Cloud Architecture

Managing these platform accounts and guaranteeing high security for the authentication process was easy to setup thanks to Amazon Cognito User Pools. When the user starts the application and logs in, an Amazon EC2 G4 instance is automatically created and used as the “ISAR server”. The connection process is automatically handled to Amazon API Gateway, so that the user directly starts his app experience without the need to setup anything.

To get custom data per user in the ISAR empowered ARES Pro Cloud application we use Amazon S3 buckets. By utilizing AWS AppSync, we were able to give our customers the ability to store their files on the Amazon S3 Bucket. This enables our frontend customer portal to be completely separated from the core infrastructure while managing all user files and also account details – a huge security aspect. And at the same time, we can access the storages from on-premise apps in our Holo-Light ecosystem, so that the files on our platform can be accessed from each of our apps in the portfolio with an easy single use login. Besides our own applications we will also empower every customer to launch their XR applications on our platform.

Furthermore, the infrastructure of AWS also empowers a high-scalable business model because of the global availability of computing resources as well as the accessibility of the marketplace and the flexible cost structure. Thanks to AWS and ISAR our customers are able to offer their XR applications in the cloud, providing a scalable, secure and attractive pay-per-use model.

More Than One Benefit

To sum it up: By rendering each application remotely as a whole, ISAR enables visualization and interaction with high-polygon, data-intensive content such as 3D objects, 3D CAD models or BIM data. CPU and GPU of the mobile device are only required to a small extent, which in the long term will help to reduce the size of the end device and optimize the form factor.

Our agnostic approach to XR devices and server infrastructure through deployed client applications also reduces the development effort for XR applications. New apps can be developed simpler and without limits and restrictions by just building a server application. Think 10-times faster app development. And forget about time-consuming optimizations to cope with limited handset performance. Above all, streaming complete applications also increases data privacy. As soon as remote rendering comes into play, the data is no longer stored on the XR device, which increases the security level drastically.

As you can tell, I get hyped by writing this with all the possibilities ahead thanks to the perfect marriage of the right cloud infrastructure and extended reality technology. Together with AWS we aim to realize the full potential of augmented and virtual reality. We want to provide more and more developers our remote rendering and streaming technology. Check out our official website and write us for a free trial access to the SDK. And in addition to the solution described above, there is also the possibility to test the Amazon Machine Image (AMI) of ARES Pro Cloud on the AWS Marketplace.