Elevenia Instills a Data-Driven Culture with AWS Analytics Platform

Until 2016, Elevenia had been running all operations and some analytics on premises using Oracle servers. “Because operations and BI shared server resources, analysts had to be conservative in running queries, especially deep analytics,” says Rangga Sobiran, head of data architecture and BI at Elevenia. “Latency was high, causing delays in returning queries, and we had to avoid downtime to the core business.”

To build a data warehouse—and, subsequently, a data lake—Rangga decided to take advantage of Amazon Simple Storage Service (Amazon S3), Amazon S3 Glacier, and Amazon Redshift. “By using Amazon Redshift, we can store unstructured data like JavaScript Object Notation [JSON], and most of our data events are in JSON format. We can store everything in Amazon S3 buckets, perform SQL queries on data stored in those buckets, and perform analytics tasks that were not possible in Oracle,” Rangga says. With Amazon S3, the team now has affordable storage with immediate procurement, a big improvement over the on-premises model. Uptime has consistently met or exceeded the 95 percent target.

“Now we have flexibility in terms of how we want to achieve transformation in the data warehouse,” Rangga says. “With the AWS Cloud, we can build more-complex analytics models that scale. Because Amazon Redshift is made for analytics types of queries, the performance is better than anything we’ve experienced.” The team has also benefited from the ease of interface with other AWS services such as Amazon Kinesis, which is used for clickstream architecture.

The diagram below illustrates the AWS infrastructure for clickstream pipeline at Elevenia: 

Shortly after establishing the data management team and infrastructure, Rangga put them to the test. Elevenia had been paying a software as a service (SaaS) provider to run the recommendation engine on its shopping site. Because the new team had some limited experience using the Python programming language, they decided to try building their own engine to compare cost and performance. Being on the AWS Cloud facilitated experimentation, and team members were eager to try new things.

“It’s difficult for us to get things right at the start, so there’s a lot of trial and error involved. This works very well in the cloud because we can create a proof of concept and run it for two weeks, and if doesn’t work, we can strike the idea out and try again,” Rangga says.

To simplify the deployment process for the new recommendation engine, the team set up Docker containers using Amazon Elastic Container Service (Amazon ECS). “In the long term, we felt that containers would make it easier for us to deploy the engine everywhere, even at scale,” Rangga says. Engineers use Amazon Fargate with Amazon ECS to automate deployment of containerized services, including clusters on virtual machines. “We really love this serverless concept because it requires minimal operations maintenance,” he adds.

After developing the recommendation engine, Elevenia compared the click rate from its homegrown engine with that of the engine provided by the SaaS vendor. Although both the click rate and conversion percentage increased by just 0.1 percent with the homegrown model, Elevenia is saving $5,000 in monthly fees by using the SaaS vendor. “It’s very significant for us, because basically we’re paying much less but getting the same return,” Rangga says.

The company has also notched substantial savings by stopping its subscription to Google Analytics Premium for tracking data in real time. Instead, it has adopted the Snowplow open- source technology, which is configured to run on top of Amazon Kinesis. With Amazon Kinesis and Amazon S3 storage supporting the Snowplow clickstream pipeline, Elevenia is now paying less than 2 percent per month compared to what it used to pay for Google Analytics. This brings its total monthly cost savings to almost $20,000 with the AWS Cloud and open-source products.

With its recommendation engine done and analytics platform in motion, the data team is focused on improving product quality. They are building a machine-learning (ML) model using Amazon SageMaker and Amazon Rekognition to assess the quality of pictures uploaded by vendors on the Elevenia shopping site. Sellers often upload low-resolution images or those with watermarks of competing online marketplaces. When completed, the ML model will assess each image and alert sellers when images are of poor quality.

Elevenia is transforming its business with the analytics platform, both in terms of cost and human resources. “Using the AWS Cloud makes it easy for my team to expand their skill set,” Rangga explains. “For people with traditional data warehouse backgrounds, coding can be pretty daunting. But with the AWS Cloud, we can save a lot of time, and it eliminates the need to learn everything from scratch.”

Feedback has also been positive from the data team, who are motivated to explore new features on AWS and learn with online videos and documentation. “It’s been a very interesting journey,” concludes Rangga. “We constantly have new ways of experimenting, and it’s been really fun.”