Announcing AWS IoT FleetWise object storage in Amazon S3

Introduction

Today, we are excited to announce that AWS IoT FleetWise now supports object storage in Amazon Simple Storage Service (Amazon S3). This new feature makes it easy and cost-effective for automotive customers to create and manage data pipelines from their vehicles. A customer can now select where vehicle data is persisted in the cloud depending on their specific use case for that vehicle data. AWS IoT FleetWise allows customers to collect, transform, and transfer vehicle data to the cloud and use that data to improve vehicle quality, electrification, and autonomy.

Automotive companies are searching for more efficient ways to simplify data collection from the vehicles. Amazon S3 support for AWS IoT FleetWise helps optimize the cost of data storage and also provide additional mechanisms to use vehicle data within a performant data lake, centralized data storage, data processing pipelines, visualization dashboards, and other improvements to downstream data services. Amazon S3 offers highly-performant and durable data management capabilities which helps with unlocking new revenue opportunities from fleets, building machine learning datasets, and creating predictive maintenance models to detect and resolve problems in near-real time. Automotive companies can use these new capabilities to gain insights on things like driving behaviors, infotainment interactions, and long-term maintenance needs for electric vehicle (EV) fleets.

Sending data from the vehicle to Amazon S3 will enable automotive data engineers and data scientists to access stored vehicle data in the format required to analyze and enrich the data. Amazon S3 object storage for AWS IoT FleetWise supports two industry standard data formats for big data implementations: Apache Parquet and JavaScript Object Notation (JSON). JSON is a standard human readable text-based format for representing structured data using JavaScript object syntax. Customers can use this format when they need to maintain relational data in the payload, though there is slight storage and compute overhead to implementing this format. Most data engineers will use Apache Parquet  format for vehicular telemetry data as it is an open source, flexible, and scalable format offering efficient data storage and retrieval. The format is suitable for data compression and encoding schemes in a variety of common programming languages.

At launch in September 2022, AWS IoT FleetWise provided Amazon Timestream as a data persistence mechanism, which is primarily built to demonstrate and analyze how data changes over time, providing the ability to identify trends and patterns in near-real time (time-series data). Amazon Timestream provides a near real-time use cases which can give, for example, fleet operators a holistic view of their telemetry data via a campaign deployed by AWS IoT FleetWise. Now, with Amazon S3, customers can unlock Online Analytical Processing (OLAP) capabilities through batch data analysis with multi-dimensional data points. This capability—switching from streaming data analytics to a more batch data processing system—allows for the identification and remediation of problems in near-real time. It also helps to continuously improve using historical data from across fleets of vehicles, creating differentiation for the operator implementing predictive maintenance in their fleet.

Data engineers can now implement tool sets using their common data processes to extract, transform, and load the data into an automotive data lake from several different sources of data, providing a centralized OLAP store for data scientists. This flexibility allows data engineers to bring vehicle data directly into other AWS services like Amazon Athena and AWS Glue, which provide abundant opportunities to enhance and enrich the telemetry data. Using services like Amazon Athena and AWS Glue also allows for formatting this data for use within machine learning models. For example, customers can continuously improve their predictive maintenance models, range estimates, or energy-based routing for EV batteries based on data stored in Amazon S3 from a battery monitoring system (BMS).

Hyundai Motor Group is innovating new features

Hyundai Motor Group (HMG) is a global vehicle manufacturer that offers consumers a technology-rich lineup of cars, sport utility vehicles, and electrified vehicles. “At Hyundai, we are focused on using the data we collect from vehicles to drive innovative infotainment features for our customers,” said Youngwoo Park, vice president and head of the Infotainment Development Group at HMG. “With more data management options available for AWS IoT FleetWise and the availability of Amazon S3, we will now be able to process batch data in addition to streaming data, giving us more ways to understand and unlock the full value of vehicle data.”

National Instruments enhances EV battery monitoring

An AWS Partner, National Instruments, will use AWS IoT FleetWise with Amazon S3 to enhance their OptimalPlus solution on AWS by building a continuous improvement data pipeline for their inference models on electric vehicle batteries. The solution allows NI’s data scientists to utilize the battery data which is aggregated from the BMS in-vehicle with AWS IoT FleetWise to continuously improve electric vehicle predictive maintenance models. These models can then be deployed to the vehicle, allowing automakers to dynamically adjust settings in the BMS to extend the remaining useful life of the battery. “Building a data ingestion and data pipeline workflow for battery monitoring systems with AWS IoT FleetWise has given us near-real time access to electric vehicle data. Now, with AWS IoT FleetWise support for Amazon S3, our data engineers will get the batched data in an extensible, flexible, and cost-efficient manner prior to bringing that data into our inference models,” said Thomas Benjamin, CTO and Head of Platform and Analytics R&D at National Instruments.

Solution Overview

Let’s take a predictive maintenance use case to walk you through the process of creating and deploying an AWS IoT FleetWise campaign that stores data in Amazon S3. Imagine you are a data scientist at a fleet operator with thousands of delivery trucks. You have the goal to lower the costs of brake system repairs and maximize vehicle uptime. To do this, you have built a machine learning model that predicts when the pads will wear out. The model requires you to gather a comprehensive dataset from various sources such as vehicle maintenance history and the type of brake pads used. However, you are missing historical data on hard-braking events that can improve the prediction accuracy. With data storage support for Amazon S3, AWS IoT FleetWise can now help you solve this problem. You will create a condition-based campaign that instructs your Edge Agent for AWS IoT FleetWise to capture 4 seconds of data before and 1 second after a hard-braking event and store it in your S3 bucket in compressed Parquet format.

Prerequisites

Before you get started, you will need:

• An AWS account with console and programmatic access in supported Regions.
• Permission to create and access AWS IoT FleetWise and Amazon S3 resources.
• To complete the AWS IoT FleetWise quick start demo to set-up the simulation and all prerequisites before creating a campaign.

Walkthrough

Step 1: Create and deploy a condition-based campaign that uploads a set of broadcast CAN signals to your target S3 bucket

1.1. Navigate to AWS IoT FleetWise console, select Campaigns (left panel), choose Create.

1.2. Configure campaign: Set the campaign name to fwdemo-eventbased-s3-parquet-gzip. 

1.3. Choose the Define data collection scheme and the Condition-based option with your individual Campaign duration. Enter $variable.`Vehicle.ABS.DemoBrakePedalPressure` > 7000 in Logical Expression and leave the optional settings as-is.

In the Advanced scheme options section, set the Post trigger collection duration as 1000 milliseconds.

In the Signals to collect section, specify the signals “Vehicle.ECM.DemoEngineTorque” and “Vehicle.ABS.DemoBrakePedalPressure.” The simulator generates a CAN message that carries the brake pedal position signal at 50 millisecond frequency. Max sample count of 100 and Min sampling interval of 0, instructs your Edge Agent to collect 5000 milliseconds of data that includes 4000 milliseconds worth of pre-event data and 1000 milliseconds worth of post-event data.

1.4. Define storage destination: Select Amazon S3.

Ensure the following bucket policy is applied to your S3 bucket (replace the $bucketName with the name of your S3 bucket).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "iotfleetwise.amazonaws.com"
        ]
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::$bucketName"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "iotfleetwise.amazonaws.com"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::$bucketName/*"
    }
  ]
}

Select Parquet as the output format with the default GZIP compression.

1.5. Add vehicles: The simulated vehicle from step 1 will show up here as fwdemo.

1.6. Review and create: Review the settings, click Create. After the status change, click Deploy to get your campaign to your Edge Agent running on your simulated vehicle.

1.7. Check data: Navigate to your S3 bucket to see your compressed Parquet files landing on the bucket every 12 to 15 minutes as AWS IoT FleetWise completes its batch write-process.

Step 2: Inspect the collected data

For business insights, you can query your compressed Parquet data with AWS Glue and Amazon Athena, and use Amazon QuickSight to visualize patterns in the hard-braking events.

Our vehicle has generated a total of 7.71K events during 11 hours of simulation. Here, we have created a simple visual that indicates a hard-braking scenario through an abrupt spike in brake pedal pressure and a drop in engine torque. Over time, this data will provide valuable historical data you can combine with other datasets such as vehicle maintenance history, brake pad type, and vehicle weight to improve the accuracy of your machine learning model.

Now, that you have verified your campaign, you can expand it to thousands of your trucks to collect more data and optimize your schedule for brake maintenance. To further increase the accuracy of your model, you can collect additional signals such as speed, harsh acceleration, or abrupt turns.

Cleaning up

Be sure to delete the following resources from your AWS account to avoid unintended charges.

1. Vehicle Simulation resources in the CloudFormation console (fwdemo stack).
2. Amazon Timestream resources with name prefixes fwdemo in the Timestream console.
3. Amazon S3 bucket.
4. Campaign in the AWS IoT FleetWise console.

Conclusion

In this post, we showcased how AWS IoT FleetWise expands the scope of data-driven use cases for our automotive customers with the newly introduced capability of sending vehicle data to Amazon S3. In addition to the near real-time monitoring and analysis provided by Amazon Timestream, the integration with Amazon S3 enables powerful OLAP use cases such as big data analysis and machine learning model training. We then used a sample predictive maintenance use case to walk you through the process of creating a condition-based campaign that collects hard-braking event data and sends it to Amazon S3.

To learn more, visit the AWS IoT FleetWise site or login to the console to get started. We look forward to your feedback and questions.