New Tools for Using Real-Time and Archived NEXRAD Weather Data on AWS

In October, we announced that the real-time feed and full historical archive of original resolution (Level II) NEXRAD data is freely available on Amazon Simple Storage Service (Amazon S3) for anyone to use. The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that enables severe storm prediction and is used by researchers and commercial enterprises to study and address the impact of weather across multiple sectors.

Early adopters have used the data to cut their product development time and ask new questions about weather-related phenomena. Today, we’re excited to share two new tools that make it even easier for you to analyze NEXRAD data and incorporate it into your workflows.

WeatherPipe for archive analysis

Before NEXRAD on AWS, it was impossible to access the full NEXRAD Level II archive on demand. This limited the types of analysis that researchers could perform with the data. Dr. Michael Baldwin, an Associate Professor in the Department of Earth, Atmospheric, and Planetary Sciences at Purdue University, recalls his difficulties with obtaining funding for NEXRAD-related research because it was deemed “technically impossible” to get enough data to perform the analysis.

He said, “As soon as I heard about NEXRAD on AWS, I got very excited about the impact for science. Having the archive available on demand on AWS opens a new world of possibilities. I’m excited to dust off that proposal and incorporate NEXRAD into my research.”

Baldwin turned to his colleague Stephen Harrell to help make it easier for students and researchers to analyze the NEXRAD data. This led to the development of WeatherPipe, an open source java tool that streamlines the process of running a MapReduce job with NEXRAD data on AWS.

WeatherPipe marshals the NEXRAD data into usable data structures and runs the job in Amazon Elastic MapReduce (EMR). The output is a NetCDF file that you can display in Unidata’s Integrated Data Viewer (IDV) and other visualization tools.

Harrell, who works in Purdue’s research computing office and is completing a degree in the Computer Science department, worked with three classmates (Lala Vaishno De, Hanqi Du, and Xiaoyang Lin) to develop the WeatherPipe prototype in a matter of weeks. They’ve open sourced it to allow anyone to use the tool and contribute to the code.

Currently, the tool produces average radar reflectivity over time. Next, Harrell and Baldwin plan to use the tool to run more advanced and specific analyses, such as storm identification and classification. Ultimately, Baldwin wants to create a predictive model for high-impact weather events, such as tornadoes.

Notifications for event-based processing

For many NEXRAD users, it’s important to get new data as soon as it’s available. This is true for both the “volume scan” archive files (the data collected by the Doppler radar site as it scans the atmosphere) and the “chunks” data (smaller packages of data that are quickly transmitted as a real-time feed).

One of the top requests from early users was for an easier way to incorporate the NEXRAD data into event-driven workflows. Today, we’re excited to announce that notifications are now available for both types of data.

We have set up public Amazon Simple Notification Service (SNS) topics for the “chunks” and archive data that create a notification for every new object added to the Amazon S3 buckets. To start, you can subscribe to these notifications using Amazon Simple Queue Service (SQS) and AWS Lambda. This means you can automatically add new real-time and near-real-time NEXRAD data into a queue or trigger event-based processing if the data meets certain criteria, such as geographic location.

Visit our NEXRAD on AWS page for information on subscribing to these SNS topics and incorporating them into workflows. We’re excited to see what you do with this new capability!

Getting started with NEXRAD on AWS

In addition to these new tools, you can find tutorials from Unidata, The Climate Corporation, and CartoDB on our NEXRAD on AWS page to help you get started using NEXRAD on AWS. Unidata has also made the NEXRAD Level II archive data available via their THREDDS Data Server and you can also browse the archive contents via the AWS JavaScript S3 Explorer.

Educators, researchers, and students can also apply for free AWS credits to take advantage of the utility computing platform offered by AWS, along with public data sets such as NEXRAD on AWS. If you have a research project that could take advantage of NEXRAD on AWS, you can apply for an AWS Grant.

We’d love to feature more tools and stories. Tell us how you’re using the data via the NEXRAD on AWS page!