Disaster response and risk management using PNNL’s Aether framework on AWS

Characterizing disaster risk and developing strategies for resilience and response to natural event hazards is core to the mission of the Pacific Northwest National Laboratory (PNNL). The science of event simulation and emergency response is based broadly on physics and earth science for climate simulations. These processes rely heavily on domain-specific data ingestion and processing for batch and continuous data sources. Comprehensive knowledge of critical infrastructure and real-time updates to assets and critical end-use loads drives the need for an agile data model capable of integrating disparate data types and maintaining an evergreen information baseline.

PNNL developed Aether as a reusable framework for sharing data and analytics with sponsors and stakeholders. Users access Aether-based projects through an intuitive web application to view and customize analytics developed by domain experts, a significant improvement over fragmented and siloed conventional methods of sharing work. Users can execute and parameterize queries and analytics through the interface and retrieve most recently available data from satellites, using secure proxy connections to publicly inaccessible databases. Data sources include Hanford Environmental Information System (HEIS), the National Weather Service, the National Hurricane Center, and others. Aether is a mature cloud-centered framework designed using Amazon Web Services (AWS) serverless services to provide a cost-effective and reliable environment for a dozen projects currently deployed with the framework. Infrastructure as code (IaC) and the AWS Cloud Development Kit (AWS CDK) allow the Aether team to rapidly deploy the framework to mission projects, accelerating time to science.

Cloud-based disaster resilience and response

One project deployed using Aether is the Electrical Grid Resilience and Assessment System (EGRASS). EGRASS forecasts the effects of hurricanes, tropical storms, and other extreme weather events and their impact on the electrical grid and critical infrastructure (Figure 1). EGRASS is used to visualize storms and their effects, deriving sequences of damage and outages with associated probabilities, and provides initial recommendations for technologies to improve grid resiliency. The framework has evolved over the last five years, and instances of Aether are deployed in various resilience and disaster response studies across the globe.

Figure 1. Screenshot showing the EGRASS storm event simulation interface. This screenshot shows a map of Puerto Rico during Hurricane Maria along with details about the storm’s strength, timeline, and probabilities for various architectural material to fail during the event.

Some other projects powered by Aether include Rapid Analytics for Disaster Response (RADR) and Chemical Security Mapping Tool (CSMT). RADR involves end-to-end satellite data processing and an early alert system for monitoring flooding and wildfire, using satellite imagery to generate analytics for structural damage, rubble and debris, downed vegetation, flooding, wildfire, and more. CSMT characterizes risks associated with mass destruction in the Philippines by developing an on-site chemical inventory by facility and uses atmospheric dispersion modeling to quantify possible plume extents and impact on nearby populations and traffic.

When moving from on-premises to AWS in 2017, PNNL designed the Aether framework to embrace AWS-managed services and serverless technologies wherever possible, largely bypassing a lift-and-shift migration. Iterating on each instance of the framework, the PNNL team developed IaC, dramatically improving time to deployment and instantiation. Deployment of a new instance now takes hours instead of weeks or months, with an average cost 60 percent lower than on-premises.

This capability has vastly improved collaboration efficiencies within PNNL and between the lab and external collaborators. Sponsors can now access research output as a dynamic product that can be calibrated and customized for specific scenarios and natural hazard events. The use of repeatable IaC allowed the PNNL team to scale the framework to many related projects and enabled a more seamless tech transfer between PNNL and external partners and sponsors.

Creating an end-to-end solution for analytic pipelines

Aether uses AWS CDK to manage infrastructure for each project, with 37 primary CDK stacks that can be included based on the project configuration and 10 companion stacks that plug in for specific environments. CDK and the serverless tools in Aether allow each project to have development, staging, and production environments with identical capabilities running at low cost.

Amazon API Gateway and AWS Lambda are the core of the Aether REST API, with thousands of API Gateway routes connected to a similar number of Lambda functions across the Aether portfolio (Figure 2). Access to those APIs is controlled through Amazon Cognito, which manages user identity and access at high granularity. Projects use AWS CodeBuild and other services within AWS Developer Tools to implement a continuous integration and delivery (CI/CD) pipeline and automatically build and deploy changes when updates are merged into development, staging, or main branches.

Figure 2. Simplified Aether architecture diagram. The major components are an API Gateway, Amazon Simple Storage Service (Amazon S3) buckets, Lambda functions, Amazon Cognito, an Amazon Aurora database, Amazon DynamoDB, a satellite data feed, AWS Fargate containers, and an Amazon Elastic Compute Cloud (Amazon EC2) instance.

Data for Aether projects is stored  primarily in Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, and Amazon Aurora. Amazon S3 provides the baseline layer for unstructured data, mainly designed using Amazon S3 Lifecycle configurations to optimize storage classes for cost and access patterns. Web applications in Aether are served up through API Gateway using Amazon S3 static hosting. Aether has a DynamoDB first policy for databases, evaluating if DynamoDB is viable and using Aurora for use cases that require a relational database. DynamoDB saves session data, simulation metadata results, application-level permission information, and other data sources with well-defined access patterns. For some projects, DynamoDB also serves as fast and inexpensive storage for time-series data, including projects that store 250 million-plus records.

Lambda functions are at the core of the computational workload. Some workflows exceed Lambda’s execution time, memory, or compute limits, so Amazon Elastic Compute Cloud (Amazon EC2) instances are used in some cases. The lifecycle of these instances is managed by Lambda functions, ensuring that instances are only running during active processing. Some third-party components, like GeoServer, are packaged as container images and run using AWS Fargate, a serverless compute engine for containers. AWS Step Functions also orchestrate Lambda and workflows based on Amazon EC2, in what would otherwise be a complex execution plan. One example of a data processing workflow is in RADR, where Lambda functions fetch satellite images and store them in Amazon S3, process them in Amazon EC2 or Fargate tasks, and then publish resulting analytics to GeoServer, where data can be consumed by the front end.

What’s next for Aether

The Aether team plans to continue abstracting the overall Aether framework and specific risk-decision modules, striving towards a more turn-key solution. Improvements will support technical transfer from PNNL AWS accounts to sponsor-owned accounts when required. The team continuously evaluates new AWS services to learn how improvements can benefit research sponsors. For example, Amazon Redshift supports the most commonly used Open Geospatial Consortium (OGC) queries at scale, and Aether plans to implement Redshift at the application and storage layers in the near future. In the realm of earth-observation remote sensing and streamlining analytics, Aether will continue to implement more sophisticated analytics at scale while adhering to a serverless-first approach.

Related stories on the AWS Public Sector Blog:

• Hurricane season 2023: Supporting hurricane response efforts with the cloud
• Understanding wildfire risk in a changing climate with open data and AWS
• Needle in a haystack: How the Pacific Northwest National Laboratory leverages the cloud to power its national security image similarity solution to better serve its customers
• Building resilience: Using technology to prepare for, respond to, and recover from the unexpected