AWS Marketplace

Using machine learning to automate tasks and increase personalization

This post describes winning solutions from the AWS Marketplace Machine Learning Challenge hackathon. Other winners created solutions using machine learning to stay connected and to support healthcare during a pandemic.

Introduction

During spring 2020, the AWS Marketplace Developer Challenge: ML Powered Solutions hackathon put forth a test for builders all over the world. The challenge was to develop a creative solution with ready-to-use machine learning (ML) models available in AWS Marketplace deployed on Amazon SageMaker. The purpose of the hackathon was to show everyday developers that they could harness the muscle of ML models with no prior ML knowledge. In this blog post, we share two winning submissions for the hackathon. These entries focus on using ML for personalization use cases, including productivity improvement and product recommendations. Winning entries focusing on using ML for connecting socially while staying at home are here.

AWS services overview

Before introducing the projects, here is a summary of the AWS services used in both solutions:

  • Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy machine learning models quickly. You can also deploy pre-trained models from AWS Marketplace in Amazon SageMaker.
  • Amazon Lex is a service for building conversational interfaces into any application using voice and text.
  • Amazon Textract is a service that automatically extracts text and data from scanned documents. Amazon Textract goes beyond optical character recognition (OCR) to also identify the contents of boxes in forms and information stored in tables.
  • Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in text.
  • Amazon Kendra is an accurate and easy-to-use enterprise search service powered by machine learning.
  • Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance.
  • Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
  • Amazon Rekognition lets you add image and video analysis to applications using proven, scalable, deep learning technology that requires no machine learning expertise to .
  • AWS Lambda lets you run code without provisioning or managing servers. You can run code for virtually any type of application or backend service—all with zero administration. Upload your code, and Lambda takes care of everything required to run and scale your code with high availability.

Winners: using ML to increase personalization

Alex: Your Personal Assistant at Work, third place winner

By Gustavo Zomer

Problem statement

Often times, office workers have to do time-consuming tasks that, while important, don’t create direct value for themselves or their organizations. These tasks might include submitting expenses, tracking to-do lists, and opening trouble tickets in issue tracking systems. Wouldn’t it be nice if we could all have personal assistants to help us with these errands? To address this personalization and efficiency challenge, Gustavo Zomer created his entry: Alex: Your Personal Assistant at Work.

Solution overview

Alex is an interactive chatbot that takes responsibilities off your plate so you can focus on what’s important to you. Gustavo Zomer built Alex over a span of 10 days with no ML experience using ML services on AWS.

Solution description

View Alex’s full architecture here. Alex uses multiple native AWS services with five different pre-trained ML models from AWS Marketplace:

You can speak to Alex through Amazon Lex, a chatbot service that provides natural language understanding (NLU) to recognize the intent of your request. Based on the intent of the text, a Lambda function identifies the workflow that can fulfill your request. Here some examples of what Alex can do:

Your situation Your request What Alex does for you
You started a new job. You must submit an image of your identification for onboarding. You say Alex, here is my passport. Alex sends the photo to the Passport Data Page Detection ML model, which scans through an image of a passport to return relevant passport details. This workflow simulates part of a standard employee onboarding experience.
You are trying to get some work done, but you can’t access the Wi-Fi. You say Alex, my internet is not working. Behind the scenes, Alex triggers Amazon Kendra, which searches through a set of documents kept in Amazon S3 to give you troubleshooting tips. If that doesn’t solve the problem, Alex uses the Mphasis Optimize.AI Expert Identifier model to determine the right support agent to deal with the service request. Alex bases this search on trouble ticket management data such as ticket priority and category.
You had lunch with a customer and must add it to your expense report before you forget. You take a photo of your receipt from a lunch meeting with a customer and say, Alex, can you add this expense? Amazon Textract pulls the text boxes from this image, and Amazon Comprehend determines identifies the relevant boxes for tracking expenses to submit for processing. Mphasis DeepInsights Address Extraction parses through the text in a receipt to return the address, to aid with automating expense reports.
While at your lunch meeting, you have a great idea for the login page of your company website. You want to remember it later. You sketch the wireframe on a napkin and say Alex, can you add a task? You submit this photo and tell Alex Add this wireframe to supplement the task.

The Mphasis Autocode WireframeToCode model takes the hand-drawn image as input and outputs the HTML code that would generate that wireframe. Then the HTML is stored in a Trello task, for whenever you’re ready to code.

 
You’re finally ready to publish your finished application. You want to look up your company’s social media policies to make sure you’re following best practices. You say Alex, find a document, and when Alex asks you what kind of document, you say social media policy. Alex again calls on Amazon Kendra to search through documents in Amazon S3. When it identifies the right file, it uses the Mphasis DeepInsights Text Summarizer model to take that text. It then returns a three-sentence summary of it for you.

You can see Alex in action here. And Alex is a friend of Alexa! Alexa Skills is integrated with Alex, so you can use any Alexa-enabled device to interact with Alex.

What’s next?

Zomer is looking to launch an Android app and web version of Alex. He’s also hoping to expand the functionalities of document scanning and text extraction and give Alex a voice using Amazon Polly.

The next winner won an honorable mention for creating an application that improves the accuracy of product recommendations at self-checkout lines.

Smart Checkout, honorable mention

by Nishir Shelat

Problem statement

Gone are the days where retailers pull the top 10 best-selling products and recommend them to their customers. Customers have thousands of products that are offered to them through various channels. Recommending the right products to the right segment is a challenging task.

Solution overview

Using few basic features, the Smart Checkout project improves accuracy of product recommendations at self-checkout lines. With no prior ML experience, Nishir Shelat used propensity models from AWS Maketplace and AWS services to build this smart application.

Solution description

The project uses following buyer propensity models from AWS Marketplace, which are offered by Prosper Insights & Analytics. These models have been developed on Amazon SageMaker based on responses from Prosper’s consumer survey.

Smart Checkout uses the image taken by the security camera at a self-checkout line and uses Amazon Rekognition to analyze personal characteristics of the person. The solution also collects additional information about the customer, such as annual income, via the application user interface (API). Using Amazon API Gateway, this information is then sent to an Amazon SageMaker endpoint. That endpoint hosts several propensity models to determine customer’s areas of interests. For example, Fashion Conscious identifies individuals for whom the newest fashion trends and styles are important. The Play Team Sports propensity model identifies whether the customer is likely to play team sports as a leisure activity. Based on these probabilities, the system displays a personalized advertisement for the customer on screen. You can review the architecture diagram and quick demo here.

Next steps

Shelat plans to extend this solution further to derive annual income using checkout history, in addition to adding sentiment analysis towards the products that are being recommended.

Conclusion

Both these hackathon projects show the creative ways developers without ML experience can apply machine learning to improve productivity and personalization to improve the customer experience. These projects were built within a span of few days by using machine learning models available in AWS Marketplace.

To find out more about other winners, view the online Tech Talk, Explore the AWS Marketplace Developer Challenge: Machine Learning Powered Solutions. This Tech Talk features Julien Simon, Principal Advocate, ML/AI and Atul Setlur, Principal Product Manager of AWS Marketplace for ML. In the talk, they explore how you can use pre-trained ML models from AWS Marketplace along with other AWS services to quickly build ML powered solutions on AWS.                

About the authors

Pallavi Nargund is a Solutions Architect with AWS. In her role as a cloud technology enabler in the Greenfield space, she works with customers to understand their goals and challenges, and give a prescriptive guidance to achieve their objective with AWS offerings.
Veena Chandran is a Solutions Architect. She loves working with her incredible nonprofit customers across the United States to help them build well-architected solutions in the AWS Cloud. In her free time, you can find her cooking and rewatching her favorite shows on Netflix for the 100th time.