How Advanced Accelerated Migrations to SaaS by Using AWS Step Functions

By Graeme Wilson, Principal Migration Architect – Advanced
By Islam Mahgoub, Sr. Solutions Architect – AWS

Advanced

Advanced Care Cloud is a software-as-a-service (SaaS) solution running on Amazon Web Services (AWS) that provides care organizations of different sizes with the power to effectively manage their entire business from anywhere, at any time.

In this post, we will cover how Advanced used AWS Step Functions to orchestrate the solution’s onboarding process and data migration activities. We’ll discuss the benefits Advanced achieved by leveraging Step Functions rather than building custom orchestration capability.

We’ll also explores how Advanced used AWS Cloud Development Kit (AWS CDK) to author and deploy all of the migration assets, providing an optimal DevOps experience.

Advanced is a British business software and services provider that operates across multiple verticals including healthcare, human resources, education, field service management, and others. An AWS Select Tier Services Partner, Advanced is also a leading international provider of mainframe migration services, having delivered more than 500 projects worldwide over four decades across all industries.

Background

Advanced wanted to be able to migrate existing customers from legacy Advanced and third-party solutions into its SaaS solution. This involves data migration activities that entail the traditional pillars of extract, transform, load (ETL) but with complex transformation of the source data before import. These legacy solutions are sometimes hosted by Advanced but are often self-hosted client-server applications deployed on customer premises.

Automation is a key requirement for this migration, as Advanced wanted to move a significant number of existing customers over a relatively short period of time. Advanced wanted to reuse existing software assets where possible, particularly as it has an extensive and validated method of importing data into the SaaS product.

The company also wanted to reuse specific parts of the customer migration process for onboarding new customers. Lastly, Advanced wanted to be able fully inspect and audit each customer’s migration, which may take several hours to complete and involves electronic documents as well as relational data, including personally identifiable information (PII) and patient data.

Challenges and Objectives

• Advanced has hundreds of customers, each with between tens and thousands of individual users, migrating to its SaaS solution over a small but ad-hoc time frame.
• The legacy software that Advanced is migrating customers from is 10-20 years old. Nothing can be taken for granted in terms of schema compatibility with the SaaS solution, and data quality is likely to be poor.
• Migration is a complex process and there are many steps often involving different teams and technologies, so for migration to be cost effective there needs to be minimal human intervention. The workflows often take longer to complete than a single shift, and the number of different skills required preclude a single person “running” all of the steps in turn.
• Additionally, automation is a bonus because Advanced needs to be able to run repeatable trial migrations for early error detection and also to create user acceptance testing (UAT) environments for their customers.

Solution Overview

In general, the legacy applications Advanced are migrating customers from are on-premises apps. They are either classic client server-based on a forms-based user interface (UI), or web-based on early .NET framework and ASP.NET with locally hosted web and database servers, typically Microsoft SQL Server.

We’ll focus on how Advanced allowed users of just one of these legacy applications to migrate to the Advanced Care Cloud solution. The same principles will be used for the other legacy applications.

Let’s start with the following diagram that depicts the high-level architecture of Advanced Care Cloud. Note that some of the details like multi-tenancy and identity management are hidden as the focus of this post is migration.

Figure 1 – High-level architecture for the SaaS solution.

Amazon Elastic Compute Cloud (Amazon EC2) instances are used for hosting the web servers, while Amazon Aurora (MySQL-compatible) is used as the backend database, and Amazon ElastiCache for Redis is used as a caching layer.

The customers’ usage of the application varies significantly throughout the day, so building an elastic architecture that scales up and down based on load would help Advanced avoid over-provisioning resources to handle business operations at the peak level of activity. Hence, reducing cost and improving Advanced’s ability to meet users demand was critical. To achieve this, Auto Scaling Groups and Amazon Aurora Serverless v2 were leveraged.

Amazon Aurora Serverless is an on-demand, auto scaling configuration for Amazon Aurora. It automatically scales capacity up or down based on the application’s needs. In this case, it allowed Advanced to run the database layer on AWS without managing database capacity.

Amazon Elastic Container Service (Amazon ECS) and AWS Fargate are used for hosting the containerized components of the SaaS solution. Amazon ECS is a fully managed container orchestration service that makes it easy to deploy, manage, and scale containerized applications. AWS Fargate is a serverless pay-as-you-go compute engine that lets you focus on building applications without managing servers.

Data Migration Requirements

Advanced’s migration workflow requires three types of data to be moved to the new platform:

• Conventional row-level data from a relational database.
• Blob data, stored in the relational database but which needs to be moved to Amazon Simple Storage Service (Amazon S3) to make it available for the SaaS solution.
• Structured XML data, stored as a Blob in the source which needs to be transformed into HTML and stored in S3.

Advanced made a decision not to migrate customers directly from their on-premises SQL Server databases, but rather to run using a backup of their data held in S3. This allows the whole migration process to take place in AWS and ensures there are no local disruptions to the migration process.

This process implies downtime for the customer, as changes to their local database cannot be made once the backup has been shipped to Advanced. This is acceptable in Advanced’s case because customers are moving from client-server software to a cloud-based solution. This necessitates some retraining for all of the users and because the switchover affects all users. At the same time, there’s a natural hard-stop to the use of the old system, followed by a move to the new software.

Customers can provide their backup over SFTP using AWS Transfer Family. This places the SQL Server backup into S3 and allows Advanced to use the “Restore from S3” function in Amazon Relational Database Service (Amazon RDS) to make the data available in SQL Server for RDS.

Migration Workflow

The simplified migration workflow is depicted in the following diagram:

Figure 2 – migration workflow.

Orchestration

Advanced decided to use AWS Step Functions for orchestrating the various activities involved in the data migration process. Step Functions provides serverless orchestration that allows customers to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines.

Contrast this with a custom-built migration solution, such as a service or command line executable, in which the business logic would be hidden to all but the developers. Process status would need to be externally visualized, and the host running the service would need permissions and networking into all of the target services.

AWS Step Functions provide a graphical view of a workflow that allows Advanced builders to discuss and visualize the process with their product and support teams. The granular detail available from the Step Functions API allows Advanced to integrate workflow status and history into their existing monitoring and management applications.

Further, Step Functions allow for orchestrating AWS services and Advanced’s own components in a simple way. There are no process boundaries to be concerned with, so a Step Function can orchestrate AWS APIs and Advanced’s own ASP.NET APIs and .NET framework command line interface (CLI) running in AWS Fargate, all in a single workflow.

Data Import

The Advanced Care Cloud solution is an evolution of an on-premises application and, consequently, Advanced has an existing, well-tested, and comprehensive data import component.

Advanced wanted to continue to use this asset rather than having to rewrite or reimplement data import, and this was made possible through containerization—the existing assets have been encapsulated in containers and deployed into Amazon ECS on AWS Fargate. The data migrations are ad-hoc in nature and run for a finite duration, making it a good fit for AWS Fargate.

With Fargate, Advanced does not have to keep EC2 instances always up and running for executing data migration activities. Instead, a Fargate container is created upon the initiation of data migration activity; it runs until the task is completed and then gets terminated. Advanced only pays for the resources requested by the migration task for the duration it has been running for.

This has been made possible by creating a command line interface for the component and then building this CLI as a Docker image that can be hosted in ECS. This task is launched from an AWS Step Function

Summary

To summarize, the steps involved in the migration are:

• The existing data import component expects CSV-formatted data for each of the business entities that are to be imported (there are over 80 of these currently).
• For row-level data, the AWS Step Function invokes an AWS Lambda function that runs SQL scripts against the source database and writes the raw, uncleansed output into Amazon S3 as CSV.
• The Step Function invokes further Lambdas that extract the Blob data by SQL script with each object also written to S3.
• On completion of the extract scripts, S3 batch jobs are used to invoke Lambdas to clean and transform the data.
• The cleansed and formatted CSV files are placed in an S3 location ready for import into the SaaS product’s MySQL database.
• The Step Function invokes an Amazon ECS on AWS Fargate task to create a container running the data import CLI.

A further benefit of using AWS Step Functions has been the ability to use AWS CDK to define and deploy the whole migration infrastructure, including the Step Function definitions. This was easier than defining, editing, and versioning state machines in JSON and allows the migration solution to be deployed from Advanced CI/CD pipelines.

In addition to the benefit of infrastructure as code (IaC), Advanced is able to quickly and safely modify and deploy changes to any part of the solution.

Conclusion

Moving customers from traditional on-premises solutions to cloud-hosted solutions is time-consuming and can be error-prone. This affects customer confidence and perceptions of quality and requires human input to ensure customer relationships are maintained and that a smooth transition takes place.

Advanced used AWS Step Functions to ensure the technical aspects of migration can be carried out quickly and repeatably, and that existing assets can be re-used. It also ensures there is a minimum of human involvement, therefore reducing planning and scheduling requirements by project teams.

.

.

Advanced – AWS Partner Spotlight

Advanced is an AWS Partner and British business software and services provider that operates across multiple verticals including healthcare, human resources, education, field service management, and others.

Contact Advanced | Partner Overview