The Internet of Things on AWS – Official Blog

Improving industrial safety with video analytics, AWS IoT Core, and AWS IoT Greengrass

Industrial customers are increasingly using the AWS Cloud to meet their targets for predictive quality, predictive maintenance, and asset condition monitoring. For more examples, see the Top Use Cases for Industrial IoT Applications ebook. The first of these, predictive quality, is often strongly correlated with the level of safety in an operating environment.

In this post, I describe a safety solution that harnesses AWS IoT Core and AWS IoT Greengrass. It was created by Bigmate, an AWS customer. Bigmate has integrated these services with preexisting CCTV cameras placed around infrastructural settings, such as warehouses, transport centers, and manufacturing sites. These AWS services equip the system to analyze, report on, and alert operators about anomalous or unexpected inputs, increasing industrial safety at the edge.

Using edge-based software and cloud-based services, Bigmate developed a platform that performs image analysis and object detection on an edge gateway. This platform can then predict an object’s path based on its previous behavior. The platform can alert operators to potential accidents and unsafe behavior.

In a typical warehouse environment, the objects being analyzed might be people and forklifts. The platform can predict and alert operators about potential collisions based on deviations from objects’ predicted paths, as shown in the following diagram.

The platform uses AWS IoT Greengrass, AWS IoT Core, Amazon DynamoDB, and Amazon EC2. The BigMate platform will soon incorporate Amazon SageMaker as well.

Bigmate initially developed the platform on standalone NVIDIA Jetson TX-2 Edge Gateways, but they quickly realized that this approach would be difficult to manage at scale across multiple facilities, edge gateways, and cameras. To enable easier scaling, they built their platform management functionality using AWS Lambda functions running locally inside AWS IoT Greengrass. This enabled the following benefits:

  • Unattended deployment of software and configurations across many environments, without the need for a developer to work onsite.
  • Direct from-the-edge integration with AWS services such as Amazon S3 to collect image samples and improve image classification model training.
  • The use of Amazon SageMaker to expedite improvement and deployment of AI models to the edge, which also expands the captured scenarios that populate the object detection library.

The Dropbear application (shown in the preceding diagram) performs multiple functions, including image capture, object detection, tracking, and depth analysis of CCTV camera streams. Dropbear is written in C++ so as to minimize latency, which is a critical factor in safety-related applications. Dropbear uses the AWS IoT SDK to communicate locally with AWS IoT Greengrass. It also uses the AWS SDK to enable image transfers to Amazon S3.

The benefit of this approach is that alerts and sirens can be triggered locally from AWS IoT Greengrass with mere milliseconds of latency. The system continues to work even without internet connectivity, ensuring that anomalies and safety concerns are detected quickly and alerts to personnel are reliable and timely.

Data is streamed to the cloud, where it is used to generate a consolidated multi-factory view on Bigmate’s custom-built visualization platform, which uses DynamoDB and Amazon EC2. If an internet outage occurs, data can be spooled and saved. When the connection is reinstated, the platform sends the data.

By analyzing data across multiple facilities, operators can run analytics to compare varying levels of safety across facilities—critical for an enterprise-wide assessment of organizational safety performance.

IoT Edge and control services

Bigmate chose the NVIDIA Jetson TX2 chipset for their edge device because of its performance with AI workloads. Bigmate’s application code—including local Lambda functions and ML inference models—can be ported to other NVIDIA GPUs. It gives the company a lot of choice and flexibility in future deployments. This flexibility is critical, considering the workloads required to detect, continuously track, and assess event generation policy, as well as calculate distance, across live video frames at the edge.

Peter Girgis, CTO of Bigmate, said that AWS IoT Greengrass helped save them 2–3 weeks of development time by incorporating AWS security services instead of building security measures from scratch. Security is a critical requirement for Bigmate and their customers because the platform manages the confidentiality and integrity of CCTV and worker safety data.

Using AWS IoT and AWS IoT Greengrass, Bigmate achieved a 15–20% reduction in development resource efforts overall, allowing the team to spend more time on the core application. By using native AWS services like AWS IoT Core and Amazon S3, they have also been able to achieve “higher overall availability.”

Bigmate quickly integrated their solution into existing CCTV infrastructure at scale by leveraging their existing video skills. There are several common protocols in use across the industry, such as UDP, RTP, and RTSP. Typically, a process can request a feed from a CCTV camera (with appropriate authentication), or a customer can configure a camera to push a camera feed to an IP address. However, video quality tuning (including jitter and bandwidth optimization) can require some experience to perform. The configuration for camera feeds is a JSON file hosted on the edge gateway. The process then deploys the configuration as an AWS IoT shadow document.

The edge gateway can connect to local devices such as sirens to alert operators to potential collisions. Bigmate tested a Sonoff SV connected to a battery in a custom 3D-printed case as an alerting device. It took approximately two hours to build this proof-of-concept. “You couldn’t do that a few years ago,” said Peter Girgis.

Machine learning

Initially, Bigmate prototyped their platform using a machine learning model based on YOLO and Darknet, mainly to speed up their time-to-market. Managing the prototype was a time-consuming manual process; it was great for early experimentation but too restrictive for production model deployments, functionality, and scaling. Bigmate has moved toward developing its own models using TensorRT, Amazon SageMaker, and Amazon SageMaker Neo. Consequently, Bigmate can now evolve its platform as problems arise, giving the company a high degree of flexibility in future deployments.

Bigmate has also implemented depth calculation for objects in CCTV frames from a single camera. Depth calculation allows the system to estimate the object’s speed and the distance between objects, making for even more accurate alerts regarding safety issues.

Lessons learned

Bigmate CTO Peter Girgis was impressed with AWS IoT Greengrass: “Fundamentally, AWS IoT Greengrass has enabled us to build our platform without having to develop the functionality ourselves.”

They have been able to rapidly develop their platform in a little over nine months with a core development team of only 3–4 people. The platform can be retrofitted and integrated into existing industrial sites. After it’s installed, it can detect accidents before they happen and provide insights about improving safety across facilities. Looking toward the future, Bigmate is planning new features that will continue to make industrial workplaces safer and more secure.

Conclusion

In this post, you saw how AWS services help the Bigmate system analyze, report on, and alert operators about anomalous or unexpected inputs, increasing industrial safety at the edge. Experiment with your own video analytics solutions, using the following posts for inspiration: