Building an AI-Based Network Infrastructure Analytics Platform with AWS

Guest post by Anand Srinivas, Co-Founder and CTO, Nyansa

Massive volumes of data are finding their way onto enterprise access networks. Analyzing and correlating all of this data across networks has become a major source of frustration for IT and network staff trying to quickly pinpoint the root cause of problems negatively impacting user and device performance.

As Internet of Things (IoT) technology pushes further into the mainstream, the problem will be exacerbated. A new set of challenges around IoT access, inventory, characterization, and security will need to be addressed.

Given the ever-growing scale of the problem, engineering a system from scratch was simply time and cost prohibitive, requiring massively scalable storage and compute power that historically hasn’t been easily accessible. Enter AWS.

AWS effectively opened the door to a better way to construct a new generation of AI-based big data network analytics systems that, until now, hadn’t been possible.

At Nyansa we quickly realized the power of leveraging AWS when developing what has now become the world’s largest network analytics SaaS solution called Voyance.

Today, Voyance is big data network analytics service specifically developed to address information blind spots cropping up within enterprise access infrastructures. Running within AWS, Voyance today analyzes hundreds of millions of client network transactions in real time from more than 17 million users and devices. That’s over 300 petabytes of data processed and counting.

At Nyansa we set out to build something that didn’t exist in enterprise IT networking—a solution to quantify network performance and device behavior from the perspective of the end user. We do this by analyzing and comparing every transaction a client makes from access to application, summarizing all the results in plain English.

But building the application was only part of the challenge. An equal amount of thought needed to be placed on the infrastructure required to power and scale it in order to maximize the amount of value and innovation the platform could deliver. Advances in mobile networks, cloud computing, big data analytics, and AI/ML offered an opportunity that previously wasn’t available.

We realized a platform approach was both necessary and possible. Voyance integrates multiple technology elements on a modern cloud infrastructure that can be extended and built upon: more data sources, services, integrations, and extensible APIs.

The Customer’s Perspective

Enterprise customers simply want technology that is easy to consume, fits with their heterogenous infrastructure, and delivers high value quickly. The SaaS model is an ideal match to address these requirements. With AWS we found the opportunity to leverage this model even further to enhance customer value while also benefitting ourselves from a development and support perspective.

Traditional infrastructure management tools, typically provide by vendors for greater visibility into specific part of the network, are often siloed and lacking the essential context necessary do deal with the variables across modern, increasingly mobile, networks. Additionally, the available data can be limited and require additional tools for aggregation and deeper analysis.

In our case the goal is to deliver the benefits of web scale and big data analytics/ML without requiring the customer to architect their own infrastructure or hire a staff of data scientists. The AWS cloud makes this possible. We ingest vast amounts of network traffic and other data sources, process it in our analytics engine, and deliver information in real time. A single source of truth across the entire enterprise network is created. IT staff receive alerts, root cause analysis for incidents, and recommendations presented visually and in plain English.

Because we have built the platform on the AWS cloud, we also have innovated in other areas that probably wouldn’t even occur to the customer because we have the platform and the data in the cloud – such as the ability to benchmark network performance across industry peers. We collect a treasure trove of data and have the analytics engine in place to offer additional services and use cases in the future. Here’s how it works:

The Nyansa Voyance Platform on AWS

It all starts with data collection. Network packets are spanned from an upstream switch to an out-of-band software collector.  This traffic includes virtually everything across the network stack such as raw data off the wire, wireless metrics via SNMP from WLAN controllers, DNS, DCHP and AAA activity from SYSLOG messages, application data from APIs as well as wide area NetFlow data from routers.

Once ingested the data is compressed, encrypted and securely transmitted over a secure sockets layer (SSL) connection to the Voyance backend analytics engine resident within AWS.  There Nyansa uses a wide range of open source tools utilizing Web-scale technologies and protocols to parse, analyze, query and correlate the data, (see figure) such as:

·  Kafka for handling and processing of real-time streams

·  Elasticsearch for full text queries

·  Redis for real-time caching

·  MongoDB for managing config and user profile data and

·  Cassandra to provide big data management of all historical data.

With all the different data variations accounted for, an Apache Spark cluster handles the majority of data processing chores.

Essential to efficiently scaling Voyance is the use of multiple different AWS compute clusters. This allows streamlining the operation and optimizing the cost of a wide range of storage and compute facilities such as those used for caching, batch data processing, historical reporting, and dynamic search functions.

Customers access their information via web-based dashboard that provides the big picture view as well as the ability to drill down to the level of a specific controller, application, or device. Over time, Voyance learns what the normal behavior of your network is and establishes a baseline level of performance for key metrics. Recommendations are automatically generated to optimize performance.

IT staff now have the information at their fingertips automatically for a range of tasks. The most obvious is for troubleshooting and root cause analysis when incidents case alerts due to deviation from ‘normal’ behavior. Additionally, the wide scope of data being analyzed is invaluable for finding problems you didn’t know existed, capacity planning, and understanding the impact of network changes.

Form the standpoint of providing a powerful and reliable service, AWS offers us sophisticated, flexible cloud infrastructure where we can meet service level guarantees for availability and security for our enterprise customers. Operationally, we can scale separate components of the analytics engine separately from each other and efficiently allocate compute and storage resources. Aggregating data at web-scale enables us to leverage the value of larger data sets to make more accurate recommendations and improve machine learning models. We can collect more data, from a wider variety of sources to analyze network performance in multiple dimensions and with multiple variables. We also realized we could do more.

Extending the Cloud to Build a Value-Adding Ecosystem

A powerful aspect of our cloud-based model is the ability to aggregate data from a large and diverse customer base. This provides the unique ability to compare and contrast different network behavior across a myriad of dimensions. Customers can benchmark their own organization against organizations of similar sizes or in the same industry. We offer a ‘friending’ feature that allows customers to agree to share data and collaborate on optimizing their networks.

The platform architecture has set the table for further extension. The breadth of IoT devices, each with their own discrete function, operating system, and protocols presents a problem that demands web scale analytics and AI. IoT also presents a unique set of security challenges, and the ability of Voyance to continuously monitor and analyze network data can provide useful insight in assessing risk, as well as detecting abnormal behavior or anomalies in order to prevent or contain attacks. Voyance can also provide data via APIs to in-house tools or other security frameworks.

Conclusion

Voyance is a fundamentally new approach to infrastructure management using AI/ML technology and big data analytics – all enabled by AWS and its scalable cloud-computing framework.  With it enterprises are able to gain quantifiable insight into the operation of their networks and the impact on end user experience and productivity – something that, until now, was never possible.