Accelerate testing with AWS Mainframe Modernization Application Testing

During the modernization of mainframe applications, testing is the most important and time-consuming phase. AWS Mainframe Modernization Application Testing is an AWS cloud-native service generally available and designed to reduce testing time and cost at scale. In this blog post, we will describe the testing approach, its benefits and show an Application Testing walkthrough.

Mainframe applications exist on a critical nexus of complexity and criticality for customers. They are typically saddled with decades of technological debt presenting challenges and hindering innovation. We hear our customers asking us to build solutions that can help with modernization of mainframe applications at a faster speed and lower cost.

We also understand that earning trust is key when modernizing mainframe applications. We regularly see customers spend 50% to 70% of their modernization project costs and time on various software testing activities which traditionally rely on very specialized skill sets. This is because testing has to ensure the modernized application has equivalent functionality and data consistency. This often becomes a challenge due to differences between mainframe and cloud services.

We recently announced the general availability of AWS Mainframe Modernization Application Testing which automates and accelerates functional equivalence testing of mainframe applications being modernized on the AWS Cloud. Application Testing provides cloud-native testing capabilities including test capture, on-demand automated test replay and comparison, and regression testing at scale.

What is AWS Mainframe Modernization Application Testing?

Application Testing is a cloud-native feature of the AWS Mainframe Modernization service available directly in the AWS Management Console. It helps accelerate mainframe application testing at scale by leveraging replay and compare automation, on-demand and cost-efficient provisioning of cloud testing resources and improved ease of use when compared to existing testing solutions.

During modernization and migration projects, Application Testing provides the ability to verify and validate that the modernized application running on the AWS Cloud is functionally equivalent to the source mainframe application.

Post modernization and migration project, you can leverage Application Testing for on-going regression testing, ensuring changes don’t break existing features throughout the application lifecycle.

What are the primary benefits of AWS Mainframe Modernization Application Testing?

Application Testing provides test case management, testing automation, testing repeatability, and scalability.

With Application Testing, you can accelerate the application test timelines, reduce testing effort and project costs with on-demand and just in time creation of target testing environments using infrastructure as code. You are now able to create and run large-scale testing projects efficiently with on-demand resource utilization including automated replay and comparison of results between the source and target workloads. Application Testing automation uses the elasticity of AWS compute resources to scale, start, stop test resources on-demand.

Testing approach

With Application Testing, we compare the functional equivalence of mainframe source and target modernized applications. Application Testing analyzes changes made to underlying data sets, database records, and 3270 screens by the source and the target application’s business logic. If the changes are identical across both the source and target application, we can assert that both source and target are functionally equivalent (Figure 1).

AWS Application Testing is architecture and language agnostic enabling support for black-box testing of applications. It brings several key advantages to our customers including increased testing efficiency, agility, speed and reduced costs.

Figure 1 – Functional equivalence testing for application through the comparison of data sets, database records and online screens

Application Testing currently supports automated comparisons for the following elements:

1. Data sets – comparing the changes between mainframe data sets and the data sets produced by the modernized application running on the AWS Cloud. This includes support for out-of-order record sequences, avoidance of false positives via equivalence rules, and automatic handling of codepage differences between source and target.

2. Databases – through an innovative State-Progress Matching (SPM) capability, comparisons of database changes done between the source and target applications occur at the level of individual data manipulation language (DML) statements (INSERT, UPDATE, DELETE) rather than comparing entire tables in their final state (Figure 2). This is accomplished by capturing database journals which contain changed records on the mainframe and uploading them on AWS to be used as reference data. On AWS, compare the database changes and the sequence of the changes done by the modernized application against the mainframe reference data.

3. Online screens – compare individual field values, length, and positions between the source mainframe application 3270 screens and the modernized web screens in the target application running on the AWS Cloud (Figure 3).

Figure 2 – Comparison of database changes between the source mainframe application and target modernized application in the AWS Cloud

How does AWS Mainframe Modernization Application Testing work?

Application Testing employs minimal distance comparison and unique state progress matching algorithms. Functional equivalence rules can be global or type specific (e.g. data set, database, 3270 screens) and highly customizable. These are applied to test suites to reduce and minimize false flagging of compared data.

Figure 3 – Comparison of TN3270 screens occur on individual field values, length and positions

Testing workflow with AWS Mainframe Modernization Application Testing

The testing workflow with Application Testing is aligned with best practices in migration projects (Figure 4). Start with the initial state of your data sets, database and 3270 screens in the source mainframe environment. Run the mainframe application and record the changes done to the underlying datasets, database records, and 3270 screens using specialized AWS provided tools. Upload these recorded artifacts to AWS from the mainframe to be used as reference data for comparison with the output data that will be created by running the modernized mainframe application test cases. The input data used by the modernized application in the test cases should be in the same initial state as the recorded mainframe data to maintain consistency.

Figure 4 – AWS Mainframe Modernization Application Testing workflow resources and objects

Application Testing automatically replays the modernized application workload (batch or 3270 transactions), capturing the changes done to the underlying data stores. After replay is complete, Application Testing compares the results produced by the modernized application during replay against the mainframe reference data (Figure 5). Any differences detected and matching data will be displayed in the Application Testing console.

Figure 5 – Reference and replay data sets, database and online screens replay and compare

AWS Mainframe Modernization Application Testing in action

Let’s now see Application Testing in action.

1. Define test case – A test case (Figure 6) is the most atomic unit of action and logically identifies the actions and business logic which will be performed on a data set, database, or 3270 screen. A test case is used to represent an independent unit of action in the application that modifies or generates output data from a given set of input data. Data changes are captured and compared at the test case level to accelerate root cause analysis of discrepancies.

Figure 6 – Defining a test case with basic information about what you’re going to test

2. Create test suite – Test suites (Figure 7) define the sequence in which test cases will be replayed and comprise of one or more test cases run in sequential order.

Figure 7 – Creating a test suite entails adding one or more test cases to the suite

3. Create a test environment configuration (Figure 8) and associate with a test suite – A test environment configuration is used for baselining the target AWS Cloud environment which will be used for replays and comparisons. This ensures repeatability and consistency across multiple testing runs. The test environment configuration includes the AWS CloudFormation template (Infrastructure as Code) that will be used to create the target testing environment and provision the resources in the same initial data state (e.g. restore of Amazon Relational Database Service (RDS) snapshot) as the reference data created on the mainframe. The test suite is associated with a test scenario and is restored for each replay of the test suite.

Figure 8 – Creating a test environment configuration involves specifying a CloudFormation template which is used to build the target test environment

4. Upload reference data – Uploading reference data (Figure 9) is the process of associating the source reference data (collected post-execution on the mainframe) to each test case in the test suite. When uploading reference data, you provide the Amazon Simple Storage Service (S3) bucket URIs where the reference data sets, database changed data capture (CDC) journals and 3270 screen capture outputs are located. These are obtained from the source mainframe using 3270 screen recorders, file transfer utilities, and database CDC tools. This is the reference data from the source mainframe that will be compared with the output of the target modernized application to validate functional equivalence between the source and target applications.

Figure 9 – Upload of mainframe reference data for comparison with target data produced during replay

5. Replay the test suite on AWS Mainframe Modernization service – Replaying a test suite (Figure 10) creates the target test environment, runs the modernized application, and captures the output data that the application creates. These are the replay data sets, database CDC journals and 3270 screen changes which Application Testing compares to the reference data uploaded to AWS from the source mainframe. The test environment created from the initial configuration is deleted after each replay which prevents it from running longer than necessary resulting in cost efficiency.

Figure 10 – Monitor the replay and comparison in real-time

6. Compare the source reference data with the target replayed data to verify functional equivalence – This is the final step where automated comparison of the source reference data and the target replay data for each test case in the test suite happens (Figure 11). If the data sets are identical or equivalent (different but should be treated as identical), the application passed functional equivalence testing. Differences in the data sets could indicate potential problems with the application business logic between the source and target and must be remediated.

Figure 11 – Dive deep into the comparison results by data set, database or 3270 screens

Go test

AWS Mainframe Modernization Application Testing accelerates the application test timelines, reduces testing effort and project costs with on-demand and just in time creation of target testing environments. It allows you to run large-scale testing projects efficiently with on-demand resource utilization including automated replay and comparison of results between the source and target workloads. To get started, please visit AWS Mainframe Modernization Application Testing documentation.

About the Authors: