Using machine learning to support healthcare during a pandemic

This post describes winning solutions from the AWS Marketplace Machine Learning Challenge hackathon. Other winners created solutions using machine learning to stay connected and to automate tasks and increase personalization.

Introduction

The AWS Marketplace Machine Learning (ML) team recently hosted a ML hackathon on The DevPost Platform, AWS Marketplace Developer Challenge: ML-Powered Solutions. A hackathon is a design sprint-like event in which software developers collaborate on projects with the goal of creating functioning software. The hackathon ran from February 19^th through April 15^th this year and had over 1,100 participants collaborating on 57 projects.

Given the current COVID-19 pandemic, many participants submitted projects that showcased how ML models can be used to develop applications that help make people’s lives safer. This blog post highlights several winning submissions that show how ML can help address healthcare challenges during the COVID-19 pandemic. See all the DevPost submissions received for the challenge.

Background

Participants used at least one ML model available in AWS Marketplace and deployed on Amazon SageMaker and at least one AWS service that is not Amazon SageMaker.

• AWS Marketplace is a curated digital catalog of listings from independent software vendors that enable you to find, test, buy, and deploy software that runs on AWS. AWS Marketplace lists pre-trained ML models and algorithms that can be directly deployed on Amazon SageMaker. For more information on how AWS Marketplace supports ML workloads, see Using AWS Marketplace for machine learning workloads.
• Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models quickly. Amazon SageMaker removes the heavy lifting from each step of the ML process to make it easier to develop high quality models.

In this post, we give an overview of four winning ML solutions:

1. Mobility Explorer traffic monitoring platform
2. Healthcam to enable personal protective equipment (PPE) compliance
3. Compare Drug Sentiments from Paper Prescriptions
4. PneumoScan to improve chest x-ray classification time for COVID-19 symptoms

Winning machine learning healthcare solutions

1. Winner: Mobility Explorer traffic monitoring platform

Submitter: Abraham Poorazizi

Overview

The winning solution for this category is Mobility Explorer, a traffic monitoring platform that analyzes mobility patterns to help monitor and enforce social distancing during a pandemic. The Mobility Explorer solution shows which areas are highly populated during a given time of the day. The following image from the solution shows a city street grid. Deep purple areas indicate high vehicle density, while bright pink areas indicate medium density, and pale pink areas show low traffic density.

Problem statement

Social distancing has emerged as an effective way to slow the spread and flatten the curve of COVID-19 infections. To help with social distancing measures, several public sector and commercial organizations are exploring applications that use mobility data. However, using mobility data for these applications has raised concerns around privacy. There are also challenges around the availability of mobility data and the dependency on telecommunications and social media companies. Mobility Explorer aims to address these challenges by using publicly available traffic camera images to identify traffic patterns and mapping them to geospatial data using ML models from AWS Marketplace.

Additional AWS services used

• Amazon CloudWatch provides you with data and actionable insights to monitor your applications.
• AWS Lambda lets you run code without provisioning or managing servers.
• Amazon RDS is a relational database built for the cloud that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases.
• Amazon API Gateway lets you create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications.
• Amazon S3 helps you store and protect any amount of data for a range of use cases.
• Amazon CloudFront a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.
• Amazon Route 53 a highly available and scalable cloud Domain Name System (DNS) web service.

Solution description

Poorazizi is collecting these traffic camera images from the City of Calgary open data API using AWS Lambda. The AWS Lambda functions are scheduled to trigger based on CloudWatch events every five minutes. AWS Lambda sends images for inference to three AWS Marketplace deep learning models:

1. Human Detector, to detect a variety of human features
2. Scooter and Bicycle Detector to detect standing scooters and bicycles
3. Vehicle Detector to detect vehicle movements and traffic patterns.

These inference results are stored in an that can be retrieved using a front-end application, built with AWS Amplify. The front-end application sends REST API requests to Amazon API Gateway to invoke AWS Lambda functions to fetch data from the Amazon Aurora data store.

• Processing layer For this solution, the processing layer begins with Amazon CloudWatch, which invokes AWS Lambda via CloudWatch events. The Lambda function sends images to Amazon SageMaker, which runs inferences on three AWS Marketplace models. Refer to the following diagram.

• Data layer The data layer then uses Aurora PostgreSQL to store the inference results used by the application. Refer to the following diagram.

• API Layer The API layer then uses API Gateway to build integrations with Lambda functions. Lambda functions include camera, summary, community summary, today trend, past week trend, and past month trend. Those functions implement the application logic and create a REST API endpoint. Refer to the following diagram.

• Presentation layer The presentation layer uses the endpoints that the API gateway creates to serve the application using a CloudFront distribution and Route53. Refer to the following diagram.

2. Honorable mention: Healthcam

Submitters:  Rico A. Beti and Stephen Mott

Overview

The HealthCam application helps organizations meet the personal protective equipment (PPE) compliance requirements and enables safety in work environments. It uses an ML model from AWS Marketplace to check if a person is wearing a mask. This model is deployed on Amazon SageMaker.

Additional AWS services used

• AWS Lambda
• Amazon Aurora PostgreSQL-Compatible Edition

Problem statement

The COVID-19 pandemic has forced many organizations to implement personal protection equipment (PPE) compliance checks to maintain safety in their environments and prevent spread of infections.

Solution description

This application is based on a Raspberry Pi Camera and utilizes PPE Detector for Laboratory Safety, which is an AWS Marketplace ML model for computer vision. This model provides predictions on whether a person is wearing a mask or not. With this information, the system is able to automatically restrict door-access to sterile areas for those people that are not wearing a mask.

There are two main components of the solution:

1. Raspberry Pi Camera with a door lock.
2. Website that gives administrators access to manually flag detection as incorrect, and to view history of compliance, violation, and manual override events.

When the Raspberry Pi camera detects movement in the frame, it sends an image along with some metadata to an Amazon API Gateway endpoint. Refer to the following diagram.

An AWS Lambda function then picks this information and queries an Amazon SageMaker endpoint created with the PPE Detector for Laboratory Safety. The endpoint returns information on the number of people in frame and the probability that each person is wearing a mask. Refer to the following diagram.

Finally, the AWS Lambda function sends a response back on whether the door should be opened or not. All the results are stored in an Amazon S3 bucket. The results from this S3 bucket can be used to query to get the list of violations and take further action for compliance. Refer to the following diagram.

3. Honorable mention: Comparing Drug Sentiments from paper prescriptions

Submitter:  Jack Sandom

Overview

This is a way for patients to familiarize themselves with the medications that are prescribed and explore alternative medications if they desire to. The solution uses a pre-trained ML model from AWS Marketplace to automatically recognize scientific drug names from prescription and look up corresponding drug review. The model is deployed on Amazon SageMaker.

AWS services used

• Amazon Textract is a service that automatically extracts text and data from scanned documents.
• Amazon Comprehend is a natural language processing service that uses machine learning to find insights and relationships in text.

Problem statement

There are many times when patients receive a written prescription from their physicians and want more information on the medications being prescribed. With this solution, patients are able to receive more information regarding their medications on their own time.

Solution description

The solution takes in the hand-written prescription and extracts the text with Amazon Textract. The extracted prescription is used to perform inference on an Amazon SageMaker endpoint created from an AWS Marketplace ML model called Drug Name NER by 7Park Data to find the specific drug names that are written on the prescription. Refer to the following diagram.

Then, the solution takes the University of California, Irvine’s (UCI) ML Repository drug review data sets, cleans and loads to Amazon Comprehend. This is used for sentiment analysis on the drug reviews from the prescribed medications. Refer to the following diagram.

The drug name identification from Drug Name NER and the drug review data set from UCI’s ML Repository are pulled together to generate a report for the patient. This report summarizes the negative or positive sentiment of the drug reviews from other patients, indications of what the drug is commonly used for, and other drug alternatives for the patient’s diagnosis. This allows the patient to consult with their healthcare providers to determine the best medications for themselves. Refer to the following diagram.

4. Honorable mention: PneumoScan

Sumbitted by Vi Ly, Sagar Bansal, Mohit Gahari

Overview

PneumoScan is a solution that uses image classification to improve chest X-ray classification time for COVID-19 symptoms.

AWS services used

• AWS Lambda
• AWS ElasticBeanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.

Problem statement

The chest X-rays of COVID-19 patients is a useful tool for physicians and radiologists seeking to verify presence of the virus. This procedure of COVID-19 diagnosis can take an average of two weeks.

Solution description

Using artificial intelligence (AI) for X-ray classification, healthcare professionals can quickly identify COVID-19 patients and reduce time to diagnosis from two weeks to under three days. This solution also facilitates face mask detection and badge authentication for medical personnel.

This team used Quantiphi Barcode/QR-code Scanner and VITech Lab PPE Detector for Laboratory Safety from AWS Marketplace. They also used an open source model COVID-Net for X-ray classification. This solution also uses Amazon SageMaker, Amazon API Gateway, AWS Lambda, and AWS ElasticBeanstalk.

This process involves an automatic badge scan and PPE kit check using a camera for authorizing a physician’s entry. After this verification, the health care worker takes the patient’s X-ray and uploads it to a web portal for X-ray classification. The portal includes an Amazon API Gateway, which invokes an AWS Lambda function to run the X-ray classification model endpoint and get the results in less than 10 seconds. If the results are classified as COVID-19 positive, they are handed over to the physician for further diagnosis and investigation.

Conclusion

Participants in the AWS Marketplace Developer Challenge accepted the challenge to develop new healthcare solutions using AWS services and AWS Marketplace ML models to help solve real-world problems. Congratulations to these winners as well as to all who submitted. To find out more about these solutions as well as what’s next for submitters, visit their solution pages.

1. Mobility Explorer traffic monitoring platform
2. Healthcam to enable personal protective equipment (PPE) compliance
3. Compare Drug Sentiments from Paper Prescriptions
4. PneumoScan to improve chest x-ray classification time for COVID-19 symptoms

You can explore AWS Marketplace machine learning models. Find out more about how to deploy them on Amazon SageMaker here.

About the authors

Sameer Goel is a Solutions Architect in Seattle, who drives customer success by building prototypes on cutting-edge initiatives. Prior to joining AWS, Sameer graduated with Master’s Degree from NEU Boston, with Data Science concentration. In leisure time, he enjoys singing, cooking, and traveling.

Han Tran is a Solutions Architect based in Seattle, WA working with state and local government customers. Han enjoys building new solutions and helping her customers. In her spare time, she can be seen outdoors and petting a dog somewhere.

Sam Palani is an AI/ML Specialist Solutions Architect at AWS. He works with public sector customers to help them architect and implement machine learning solutions at scale. When not helping customers, he enjoys long hikes, unwinding with a good book, listening to his classical vinyl collection and hacking projects with Raspberry Pi.

Srinath Godavarthi is a Sr. Solutions Architect at AWS based in the DC metro area. He has over 20 years of experience in the IT industry and helps public sector customers achieve their mission objectives with well-architected solutions on AWS. Prior to AWS, he worked with various systems integrators in healthcare, public safety, and telecom verticals. He focuses on innovative healthcare solutions using AI/ML technologies.