Finding Value in Your Digital Analytics Data Using Analytics Shift with Softcrylic and AWS

By Francis Lavelle, Sr. VP Data & Analytics – Softcrylic
By Kenny Rajan, Sr. Solutions Architect – AWS
By Anuj Dewangan, Principal Solutions Architect – AWS

Softcrylic

Continuous streams of data are exponentially generated by online platforms. The value of digital analytics data is many fold, as customers provide behavioral indicators and preferences in how they interact with online applications.

Businesses are beginning to leverage this digital data in advanced analytics and data science models to better understand customer needs and unlock new opportunities for growth.

Digital clickstream data can be difficult to decipher—a common problem is determining the best way to bring online digital data into data warehouses stored in Amazon Web Services (AWS).

There is a need to determine how to model the data from various sources—including websites, mobile applications, kiosks, over-the-top (OTT) platforms, and more—to obtain greater insights into areas where they need improvement, such as the following:

Attributing value to website traffics sources and marketing channels, like search, paid media, and social.
Content performance by user segment and contribution rate to conversion or lifetime value of an article.
Customer performance across digital and offline channels.
Digital customer data integration with traditional customer relationship management (CRM) data on the demographics, sales, loyalty, and geo.

Using digital analytics solutions provides a framework for collecting the online data which is captured for every page load or click event on company websites and mobile apps. These solutions are incredibly valuable and help organize how to track behaviors and measure events to identify what’s working for your company and customers.

Many leading enterprises leverage digital analytics solutions, like Adobe Analytics, a global leader in digital analytics. Its tag management and analytics platform lets you mix, match, and analyze data from different touchpoints in the customer journey and get versatile reporting with predictive intelligence.

Although the business case for digital analytics solutions is well-articulated, businesses may look for ways to build stronger cases around transformations by consolidating data generated across enterprise with customer behavioral data. They may also look for ways to activate internal analytics tools to dive deeper into customer behavior data for even greater insights using scalable and high-performing analytics platforms on AWS.

In this post, we will walk through how Softcrylic, an AWS Select Tier Services Partner, developed the Analytics Shift solution on AWS Marketplace. Analytics Shift helps businesses bring Adobe Analytics data into Amazon Redshift to drive deeper insights and data integration.

Driving Value with Your Digital Analytics Data

Analytics Shift allows companies to quickly bring their hit level digital data into a manageable dataset and analyze individual customer journeys and interactions with web properties.

With Analytics Shift, you can build machine learning (ML) models for customer segmentation, attribution, and other applications. Unifying your Adobe Analytics data with your CRM, point of sale (POS), and other data sources is now possible. It also enables you to accelerate access to your online data—with no more waiting for data warehouse or clickstream export—and back up your customer analytics data indefinitely.

Granular analysis of customer behavior data from solutions like Adobe Analytics with other data can be tedious using out-of-the-box solutions. Analytics Shift addresses this gap by opening an organization’s universe of customer behavior data to direct queries by analysts and marketing managers in a familiar SQL-based environment.

Analytics Shift empowers organizations to provide deeper insights more quickly, as well as explore new avenues of data activation. Analytics Shift helps customers get the most out of their Adobe Analytics data.

Key Benefits of Analytics Shift

Granting easier access to granular customer behavior data and unifying the customer behavior data with other enterprise data for deeper insights.
Access to queryable customer behavior data from Adobe Analytics in Amazon Redshift with hourly or daily refreshes.
Backup and long-term storage of web analytics customer data.

Figure 1 – Opportunities in Analytics Shift.

High-Level Data Activation Workflow

Analytics Shift offers an efficient and flexible solution for bringing your Adobe Analytics data into Amazon Redshift. Onboarding involves the following steps:

Build a customized data model.
Process raw Adobe data for SQL access.
Automated load of data.

The goal of Analytics Shift is to make the process of modeling your analytics data easy and efficient for customers. It also manages data ingestion in a software-as-a-service (SaaS) model, even if you have hundreds of raw files generated daily consisting of greater than 1,000 columns.

Businesses can work with Softcrylic to customize a data model tailored to their unique business needs. After the data model is configured, Analytics Shift data connectors begin processing Adobe Analytics raw data that can be stored in Amazon Simple Storage Service (Amazon S3). Then, the data is processed using an Amazon EMR cluster. Finally, clean modeled data is pushed into Amazon Redshift for use.

Once the implementation is complete, Analytics Shift loads the data into the data models either daily or hourly (depending on the customer needs) in a completely automated and transparent fashion.

With Analytics Shift, customers can integrate Adobe Analytics data with other internal datasets quickly. It also enables access to audience-level data, unlocking the ability to analyze customer journeys, model audiences for attribution, improve identity stitching, build machine ML models, and more.

Analytics Shift Architecture

Analytics Shift is architected to automate the ingestion of Adobe Analytics data into Amazon Redshift, as shown in the data flow diagram below.

Figure 2 – High-level architecture for Analytics Shift.

Architectural Overview

Data sources: Represents the data that will be ingested into the platform. This includes variables (props/evars), traffic sources, shopping activity, product views, visit frequency, page visits, campaigns (social, email, SMS), tracking codes, marketing channel conversions, platform data (mobile, web, app, OTT), and success events.
Data ingestion: Data will be collected using batch process. Businesses can choose Amazon S3 as landing zone to store the Adobe Analytics data with SFTP and FTP integration using AWS Transfer Family.
Data processing: Data transformation and processing is done using dynamically created EMR Spark cluster.
Data platform: The data collected goes through the processes outlined below to become ready for activation. The platform stores all collected data in a centralized data lake, and the collected data is eventually loaded into appropriate schemas in a data warehouse (Amazon Redshift).
Data tools: Various tools can be deployed on top of a marketing warehouse that allows users to deliver a consistent data story to executives, analysts, marketers, and research teams from single source of truth.
Driving value with data using AWS services:
- Machine learning models (Amazon SageMaker): The data can be used for both predictive and descriptive modeling. Examples of use cases include persona generation, demand forecasting, and ad attribution.
- Graph database (Amazon Neptune): The data can be loaded into a graph database along with other data sources for ID stitching, finding new insights, and more.
- In-memory database (Amazon ElastiCache): Customer data traits or profile data can be loaded to an in-memory database and made available to optimization tools which can personalize site experiences through real-time message and offer decisioning based on personas and site behavior.
Activating data governance council:
- Define and manage data access policies for users, vendors, and third-party tools.
- Monitor and audit all data access and usage to ensure data breach.
- Define disaster recovery process and procedures.
- Enable appropriate data compliance standards are met based on country requirements.

How Analytics Shift Helps Cross-Functional Teams

With Analytics Shift, cross-functional teams—digital and marketing teams, for example—can extract customer behavioral data, tracking codes, and product information using Adobe Analytics data. They can also transform and aggregate the data with enterprise datasets according to specific business requirements, and then load the data into desired landing zones, such as enterprise data warehouse solutions.

With these meaningful insights gathered, businesses can tap into opportunities, such as improved attribution of marketing information based on POS and other enterprise data. You can also deploy sophisticated customer segmentation and simplify the process of consolidating multiple views of your customer to have a consistent experience across the organization.

Moreover, you can import the data with additional context back into Adobe Analytics to enrich the report suite. See Adobe documentation to learn more about import options.

Analytics Shift Solution Features

Quickly solve your data integration and modeling issues and unlock the power of your digital analytics data. Key features of Analytics Shift include:

Data collection and integration: Unified data at the customer level for identified customers, providing views across customer behavior to enable pathway development.
Data preparation: Automate the data preparation of millions of customer interactions across all marketing and direct referral channels.
Data science: By running a segmentation analysis, customers can identify known member clusters and generate insights for audiences across marketing channels.
Multi-touch attribution: Using customer data from online properties and paid media, customers can leverage open-source models based on Markov chain and pathways analysis.

Figure 3 – Analytics Shift services.

Summary

For many companies, user groups such as data engineers, marketers, frontend developers, executives, and external partners require access to different kinds of data from different sources.

Businesses are looking ways to unify digital analytics data from solutions like Adobe Analytics with other data generated across enterprise applications to get deeper insights about the customer across their digital journey.

Softcrylic can help you simplify these integrations so that you can focus on what matters the most—gathering information on the performance of your online platforms and determining if your business aims are in line with customer needs. To learn more, please contact info@softcrylic.com.

.

.

Softcrylic – AWS Partner Spotlight

Softcrylic is an AWS Select Tier Services Partner that developed the Analytics Shift solution which helps businesses bring Adobe Analytics data into Amazon Redshift to drive deeper insights and data integration.

Contact Softcrylic | Partner Overview | AWS Marketplace

*Already worked with Softcrylic? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog