Performance Testing in Continuous Delivery Using AWS CodePipeline and BlazeMeter

Editor’s note: Visit the AWS CodePipeline and BlazeMeter websites for the latest information.

This is a guest post from our friends at BlazeMeter, an APN Technology Partner

By now, most software delivery teams have heard about and are either practicing or planning to practice some flavor of continuous delivery. Its popularity has exploded in recent years largely because it has proven to have immense benefits for the rapid release of high-quality software. After each commit, the software is built and tested, and a deployable artifact is the result. How or when that artifact is deployed to either a staging area or to production depends on the team, their process, and their infrastructure.

While unit and functional tests have become standard practices of good software delivery, load and performance tests have been a bit neglected in many workflows, reserved for specially scheduled events and generally conducted manually by a group. In part, this is because load and performance tests have tended to involve complex and brittle scripts that require dedicated, vendor-specific environments and are difficult to automate or run quickly enough for fast feedback.

Since AWS CodePipeline is such a powerful automation framework for managing the continuous delivery process from start to finish, let’s take a look at how we can more easily inject automated load tests at the right places in the delivery workflow with BlazeMeter’s native AWS CodePipeline integration.

Who is BlazeMeter?

BlazeMeter, based in Mountain View, CA, provides an easy-to-use, cloud-based performance testing platform that can be accessed directly from any stage of AWS CodePipeline (as a Test action) at any point where load, stress, or performance tests need to run. BlazeMeter extends Apache JMeter technology by providing some important pieces, like automatic scaling and professional results reporting. If your team hasn’t already adopted JMeter, it’s a very powerful and flexible open source tool capable of orchestrating any type of performance test, from the simplest to the most sophisticated. If you are already using JMeter, you can begin working with BlazeMeter right away.

What Kinds of Performance Tests Should We Run?

When it comes to performance tests in the delivery pipeline, different architectures and objectives call for different strategies.

For example, if you’re deploying an API server that handles a lot of incoming requests from mobile devices or other applications, tests might focus mostly on throughput: the hits or requests per second that various endpoints can handle within given response-time expectations. Those tests can use straight URL requests without regard for the complexities of think time or extraneous logic that synthetically shape traffic.

To perform this type of test in AWS CodePipeline, Edit the Pipeline, add an Action to the target stage where the API test should run, choose Test as the Action category, and choose BlazeMeter as the Test provider.

After choosing Connect, you’ll be taken to BlazeMeter’s sign-in page. If you’re not already a BlazeMeter user, you can create a free account right there and have instant access.

Next you can choose New API Test from the different types of tests BlazeMeter offers.

BlazeMeter provides an easy-to-use utility where you can simply enter your endpoint URLs and required payload data. You can add the URL, specify the HTTP verb (GET, POST, PUT, DELETE), and even add custom headers. In this example, I’m providing the necessary Content-Type header as well as a JSON payload for my POST request that will test selecting cities in a flight reservation app.

Two things to take special note of in the test configuration:

– Amazon CloudWatch integration. Here you can have BlazeMeter include Amazon CloudWatch metrics for your Amazon EC2 instances involved in the test.

– Thresholds. Use this feature to define what will constitute test failure, such as average response time or percentage of errors being above selected values.

API-oriented test scenarios like these could run immediately after an AWS CodePipeline action that uses AWS CodeDeploy or AWS Elastic Beanstalk to configure a staging environment, and they can run quite speedily.

Simulating Realistic Traffic in Automated Load Tests

A more thorough and real-world performance test will take a little more time to set up and will require Jmeter scripts. Rather than just hitting the app with a barrage of HTTP requests, we want to be more strategic in how we shape the overall load profile. (Getting started with creating Jmeter scripts is a bit beyond the scope of this blog post, but we’ll provide some useful tips below.

For these more realistic tests, we once again add an Action to the desired stage in AWS CodePipeline, and choose BlazeMeter as our Test provider, but this time we’ll select New Jmeter Test.

Business considerations enter the picture at this point. How many users do we expect? What will they be doing with the app, and how frequently? It’s often useful to include business analysts and product marketing teams in these discussions as they can bring useful metrics about user activity.

For these scenarios, we should create Jmeter scripts that represent different types of expected interactions. For example, if we’re testing a flight reservation website, we should have some users browsing and looking at flight prices, while others are making reservations, and still others are canceling flights or choosing hotels. And since humans stop to read pages or fill in web forms, we should make use of scripted timers, such as Jmeter’s Uniform Random Timer, to introduce those natural delays into the test.

Ultimately we want to understand what we sometimes call “business throughput”: How many successful actions customers can perform, how many search results are returned, or how many total flights are reserved. Choke points and constraints around these items have a direct impact on the business so they tend to be the important elements to focus on during the test. Also, since we know the underlying components of the stack involved in these transactions, this data gives us ideas about where to start our investigations.

Using Jmeter’s Transaction Controllers and naming them clearly will help you identify these business transactions after the test run.

In the example below, I’ve labeled different actions in a flight reservation system and the BlazeMeter report tells me about response times and number of transaction calls.

Let AWS CodePipeline Do the Work

Now that we can automate any kind of performance and load test using AWS CodePipeline and BlazeMeter, we hope to help teams focus on the more critical tasks of fixing defects and optimizing and tuning the bottlenecks that these automated tests discover. Since tests run so frequently, baselines start to develop and we can observe trends that provide a sense of familiarity with how our apps behave. Tuning gets easier, and aberrations become more evident.

Before you know it, you’ll be confidently releasing to production without thinking twice, knowing that your users are seeing high-class performance.