Unleashing sustainable potential: harnessing machine learning on AWS for optimal energy usage

Organizations have been looking to modernize their legacy Building Management System (BMS) with data analytics and artificial intelligence/machine learning (AI/ML). Although they can help organizations achieve their sustainability goals, improve energy management, and other smart building capabilities, they can also turn into a substantial setback if the systems aren’t set up properly to use the data and analytics. Users may have difficulty in getting different data systems to talk to each other, and as a result may lose out on the values of the collected data.

In this post, we walk through a high-level reference architecture and demonstrate a reinforcement learning (RL)-based solution on how organizations can use Amazon Web Services (AWS) services to optimize energy consumption in their buildings. In our case study, we achieved reduction in operational costs by 5% ($2.5 million cost reduction in the first year) along with its associated carbon emissions. If you’re interested in adopting this solution, then you can find the reference architecture, and the open source GitHub implementation.

The Guidance for Monitoring and Optimizing Energy Usage on AWS

Organizations have faced challenges in managing building operations and optimizing energy efficiency within their BMS. Data silos can hinder a comprehensive view of energy consumption patterns and identifying areas for improvement. Multiple vendors and their unique sensor naming conventions can complicate data integration and cross-environment analytics. AWS has introduced the Guidance for Monitoring and Optimizing Energy Usage in response to these customer challenges. In this post we dive deeper with a detailed architecture that allows the developer to understand the holistic implementation approach, as shown in the following figure.

Figure 1: Reference architecture for implementing energy optimization on AWS

The following is a detailed walkthrough of the reference architecture:

1. Use AWS IoT Greengrass Core to connect, publish, and subscribe your own building data using open standard MQTT or LoRaWAN protocol. This allows you to connect your facility sensors to the AWS Cloud.
2. Use AWS IoT Core to connect to AWS Cloud, manage your devices, update over-the-air (OTA), and secure your devices. You can also use it for connecting and managing your asset metadata and other essential supporting data.
3. Employ the Internet of Things (IoT) Rule Engine to separate your batch and real-time messages. Process your real-time messages with Amazon Kinesis Data Streams and batch messages with Amazon Kinesis Data Firehose, encrypted with AWS Key Management Service (AWS KMS). Route real-time messages to AWS IoT SiteWise Real-time datastore and batch messages to the landing zone of your Amazon S3 data lake.
4. Build your data lake to store raw data and create curated processed data in Amazon S3 using AWS Glue. You can trigger AWS Glue periodically or on-demand to process your raw telemetry data into the format needed for downstream analytic use cases.
5. Use Amazon Neptune and AWS IoT SiteWise to enrich your IoT sensor data with Asset Model and Model Hierarchy.
6. Use Amazon SageMaker for building, training, and deploying the Energy Optimization model at scale. Use SageMaker Pipeline to orchestrate the end-to-end ML workflow as an alternative to AWS Step Functions, and SageMaker Training for training the model as an alternative to AWS Batch. Analyze the collected sensor data using the SageMaker built-in algorithms or custom models to develop predictive models, and store the trained model as a Docker image in Amazon Elastic Container Service (Amazon ECS).
7. Build a real-time operational dashboard using AWS AppSync and AWS Amplify, which allow you to monitor your energy usage. Alternatively, you can use Amazon QuickSight or Amazon Managed Grafana for dashboards to identify areas of improvement, provide recommendations to address high-cost operations, and gain a holistic view of your facility operations.
8. You and your users can interact with the real-time dashboard with federated authentication through Amazon Cognito, which allows granular controlled access patterns.
9. Deploy ML-based recommended IoT setpoints through SageMaker. Optionally, you can deploy edge models on AWS IoT Greengrass Core.

RL energy optimization solution

Intro to RL
RL is a subset of ML distinct for its focus on enabling agents to make sequential decisions within an environment to maximize cumulative rewards. In our context, RL serves as the foundational framework for optimizing the energy consumption of a rooftop unit (RTU) system. Buildings account for a substantial portion of global energy use, with heating, ventilation, and air conditioning (HVAC) systems being among the largest contributors. By using RL, our approach adapts to geographic-specific weather conditions, outperforming traditional methods like those offered by the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) standards, which provide guidelines and best practices for HVAC system design and operation. This adaptability enables building owners to significantly lower energy consumption, achieve meaningful cost savings, and reduce their environmental impact. RL achieves these goals by continuously learning and updating its control policies through interactions with the environment, adapting to dynamic conditions and optimizing for specific objectives like energy efficiency and occupant comfort.

Data
The RL environment heavily relies on historical data that forms the backbone of our model’s training and the RTU system behavior. This historical dataset is a collection of timestamps, encompassing a multitude of outside and return air properties, such as temperature, humidity, enthalpy, and other relevant parameters. AWS IoT Greengrass Core and AWS IoT SiteWise Edge collect the data from the edge and use AWS IoT Core features to bring data into the cloud as mentioned in the preceding architecture. The data flows into Amazon Kinesis for processing, which culminated a year’s worth of historical data residing in Amazon S3. This refined data formed the cornerstone for training our RL agent, thereby furnishing the vital information necessary for effective learning and optimization.

Model approach

The Soft Actor-Critic (SAC) RL algorithm stands out due to its use of a stochastic policy, enabling exploration of diverse action choices and stopping the agent from being trapped in local optima. What sets RL apart is its suitability for training individual RTUs to make real-time, adaptive decisions based on their geospatial locations. This real-time adaptability is a limitation in traditional ML, which relies on static datasets and predefined rules.

The SAC algorithm combines policy-based (Actor) and value-based (Critic) RL methods, making it well-suited for building optimization tasks. It uses state-value and action-value functions to estimate long-term accumulated rewards, along with an advantage function to measure the benefit of actions relative to the average expected value at a given state. The Actor’s policy neural network is updated using policy gradients to maximize rewards, while the Critic’s neural network is refined to better approximate the true value function. Importantly, SAC introduces an entropy regularization term, encouraging exploration and balancing expected rewards with entropy.

Figure 2: Illustration of an RTU equipped with an economizer

Figure 3: Psychrometric chart for economizer operation

State: The state space includes outdoor temperature, outdoor humidity, return temperature, and return humidity. These are crucial parameters for air ratio calculation and enthalpy interpolation. We assume supply air temperature and humidity to be constant, as specified in the following Table. Figure 3 shows an overview of an economizer operation. The economizer reduces the load of cooling on the system, and we model how much energy can be saved with free cooling using available sensor measurements. Our economizers are equipped to monitor outdoor temperature and humidity, and adjust the damper position based on a setpoint. The damper position is kept at a minimum level of 10% open to allow for adequate fresh air ventilation. Maintaining a damper opening of at least 10% is the industry-accepted practice to maintain proper air exchange and introduction of outdoor air into the building. Economizers effectively reduce the cooling load by minimizing the enthalpy required to reach the desired supply air temperature setpoint. Figure 2 shows the psychrometric chart for economizer operation, which captures the relationships between temperature, humidity, and enthalpy.

Table 1: Constants in the RL environment

Action: The action space consists of two continuous setpoints: Economizer maximum temperature and Economizer maximum enthalpy. These setpoints are constrained within specific ranges: [40, 75] °F for temperature and [20, 30] BTU/lb for enthalpy. Actions are selected hourly, allowing for frequent adjustments to optimize performance.

Reward: The reward function is designed to estimate the power required to achieve the desired supply air temperature, considering the economizer status. It uses a modified Total Heat Gain equation and penalizes high differences between entering and leaving air enthalpy, indicating longer mechanical cooling run times. The reward is the negative (or inverse) of the power consumption. This means that a higher reward value corresponds to lower power consumption, and a lower reward value corresponds to higher power consumption.

This approach allows the system to effectively balance the use of free cooling from outside air with mechanical cooling, maximizing energy efficiency while maintaining optimal comfort levels. By using these state observations, action selections, and reward calculations, the RL agent learns to fine-tune economizer setpoints based on specific environmental conditions and locations, which is crucial for optimizing HVAC systems across varying climates, humidity levels, and local regulations.

Results

Figure 4: Comparison of Power Consumption with Different Economizer Setpoints

The RL agent’s performance is evaluated in terms of power consumption reduction. The provided information suggests that the RL agent has achieved up to 5% reduction in power consumption leading to substantial cost savings of $2.5 million in the first year of its deployment, while maintaining the same comfort level, as compared to the ASHRAE Standard 90.1-2013 setting. This indicates the effectiveness of the RL approach in optimizing the economizer settings of the RTU system.

Conclusion

HVAC systems play a crucial role in making sure of indoor comfort within buildings. However, these systems often contribute to significant energy consumption, leading to high energy bills and increased carbon emissions. As global temperatures continue to rise, the need for effective cooling is growing, making the optimization of RTUs increasingly important.

One way to achieve this optimization is by focusing on the economizer component within RTUs. Building owners can harness the potential of temperature and humidity sensors alongside state-of-the-art ML algorithms to fine-tune economizer usage. This not only reduces energy consumption but also makes sure of an optimal indoor environment.

AWS has a comprehensive suite of cloud services and tools designed to assist building owners and operators in implementing intelligent HVAC systems. For a comprehensive exploration of our experiences and challenges of training and deploying a cloud-based RL agent across more than 10K+ RTUs at over 200+ sites, you can find the details in our publication. We also offer prescriptive guidance on best practices and targeted business outcomes along with addressing frequently asked questions about adopting this solution.