AWS Cloud Operations Blog

Automating Feature Release using AWS AppConfig Integration with AWS Codepipeline

Last year, we released AWS AppConfig a new capability within AWS Systems Manager to create, manage, and quickly deploy application configurations. AppConfig enables you to validate your application configuration before deployment and enables you to deploy configuration in a controlled and monitored way.

AWS AppConfig enables you to deploy configuration changes independent of the application code deployments. This means that your applications are not required to be restarted or taken out of service, whenever an application configuration is updated. With AWS AppConfig, configuration updates are immediately available to the application. You can use the AWS AppConfig API, AWS CLI or AWS SDK to retrieve configuration updates at runtime.

Since its launch, we have seen customers adopting AWS AppConfig for variety of use cases, with one of the top use cases being the ability to release new features independent of code deployments. Application configuration deployments are faster than code deployments as configuration files don’t require a build stage and can be deployed at runtime without taking the application out of service.

Often, these feature releases require coordination among multiple teams and manual processes to correctly order the configuration of backend services and front-end customer experiences. With coordination and manual deployments, teams run into risks of delays, which affect the end customer experience.

To simplify the manual task of deploying application configuration across multiple environments, applications and Regions, we are announcing the integration of AWS AppConfig with AWS CodePipeline. With this launch, we enable customers to adopt to best practices used by thousands of teams within AWS by easily decoupling their code changes from feature releases and automate the release of these features in a safe and efficient manner.

As a part of this integration, we are adding the following functionalities:

  • New configuration source – AWS CodePipeline is now a new configuration source in AWS AppConfig configuration profiles. This complements the existing four configuration sources – Amazon S3 object, AWS Systems Manager document, AWS System Manager parameter and recently released AWS AppConfig hosted configuration.
  • New deploy action provider – AWS AppConfig is now a deploy action in AWS CodePipeline.

Following code deployment best practices, storing configuration in a source control management system like Git enables you to create a branch and keep track of version changes. When changes are critical, a manual approval action helps mitigate risk. A separate pipeline for configuration deployments therefore not only makes the release process automated, but also reduces the chances of errors caused by manual deployments.

In this previous AWS AppConfig blog post, we saw how we can deploy application configuration from the AWS Management Console.

In this post, we explore how we can leverage this new integration between AWS AppConfig and AWS CodePipeline to quickly build a full-fledged CI/CD pipeline to automate configuration deployments to multiple environments.

Example application overview

To illustrate this new feature, we work with a simple serverless REST API that returns a sample list of AWS services. The application comprises of Amazon API Gateway and AWS Lambda. To simulate the change release flow, we work with three environments: development, testing, and production environments.

We introduce a new feature to the API that returns a limited number of AWS services based on the feature parameter. The feature name and the feature parameter will be stored in AWS AppConfig. AWS Lambda code is updated to read the application configuration from AWS AppConfig.

We will then build an AWS CodePipeline, to seamlessly deploy the application configuration to multiple environments in an automated manner, without a need to build and deploy application code.

The following diagram illustrates the solution design and the AWS CodePipeline flow:

 

 

Step 1: Create base application stacks using AWS CloudFormation

To set up the base application, which includes Amazon API Gateway, and AWS Lambda, we are using CloudFormation.

  1. To begin, use the following code block to save a template via local file in your computer (.YML file extension).
  2. Then open CloudFormation console, and click on “Create stack”, selecting “With new resources” option.

In the next screen, under the “Specify template” section choose “Upload template file“, and provide the file you just saved.Click next, give the stack a name like “ListServices-dev“, and choose development as value for the Environment parameter.Click next, optionally define your tags, and click next again.  On the last screen, don’t forget to tick the three check boxes in the “Capabilities and transforms” section, and finally click on “Create stack” button.

Repeat this process for the remaining two environments – testing and production. Specify ListServices-test and ListServices-pro as stack names and testing and production as the Environment stack input parameter. Wait until the stacks are in create complete status.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: AWS SAM template for AWS AppConfig-Codepipeline integration demo
Parameters: 
  Environment: 
    Type: String
    Default: development
    AllowedValues: 
      - development
      - testing
      - production
Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: 
        Ref: Environment
  ApiFunction: # Adds a GET api endpoints at "/" to the ApiGatewayApi via an Api event
    Type: AWS::Serverless::Function
    Properties:
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /
            Method: get
            RestApiId:
              Ref: ApiGatewayApi
      Environment:
        Variables:
          appEnv:
            Ref: Environment
      Policies:
        - Statement:
          - Sid: GetConfig
            Effect: Allow
            Action:
            - appconfig:GetConfiguration
            Resource: !Sub 'arn:aws:appconfig:${AWS::Region}:${AWS::AccountId}:*'
          - Sid: ReadS3
            Effect: Allow
            Action:
            - s3:GetObject
            - s3:ListBucketVersions
            Resource: !Sub 'arn:aws:s3:::codepipeline-${AWS::Region}-*'            
          - Sid: WriteLogs
            Effect: Allow
            Action:
            - logs:CreateLogStream
            - logs:CreateLogGroup
            - logs:PutLogEvents
            Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*'
      Runtime: nodejs12.x
      Handler: index.handler
      InlineCode: |
          exports.handler = async (event) => {
            let result = getServices();
            let appEnv = process.env.appEnv;
            const response = {
              statusCode: 200,
              body: JSON.stringify(result),
            };
            return response;
          };

          function getServices() {
            return [
              {
                name: 'AWS AppConfig'
              },
              {
                name: 'Amazon SageMaker Studio'
              },
              {
                name: 'Amazon Kendra'
              },
              {
                name: 'Amazon CodeGuru'
              },
              {
                name: 'Amazon Fraud Detector'
              },
              {
                name: 'Amazon EKS on AWS Fargate'
              },
              {
                name: 'AWS Outposts'
              },
              {
                name: 'AWS Wavelength'
              },
              {
                name: 'AWS Transit Gateway'
              },
              {
                name: 'Amazon Detective'
              }
            ];
          }
Outputs:
  APIGateway:
    Description: The API Gateway exposing your testing app
    Value: !Sub "https://${ApiGatewayApi}.execute-api.${AWS::Region}.amazonaws.com/${Environment}"

Test the created applications, by browsing the endpoint in the outputs tab of the CloudFormation stacks. It should return a JSON with a list of sample AWS services.

Step 2: Create application, environments and configuration profile in AWS AppConfig

1.     To get started, go to the AWS Systems Manager console and select AppConfig from the navigation panel on the left.

2.    On the AppConfig console, click Create Configuration Data and specify the name of the application. You can add an optional description and apply tags to the application.

 

3.    Once the application is created, you are redirected to a page with Environments and                Configuration Profiles tabs. Let’s start by creating an environment by clicking Create environment and specifying the environment name and optional description.

4.    You can also optionally add tags and configure monitors, that is, Amazon CloudWatch alarms for each environment. Let’s create three separate environments as shown below.

5.    Next, let’s set up a configuration profile. Select the Configuration Profiles tab, and click Create configuration profile.

Provide the name of the configuration profile as ListServicesCodePipelineConfigProfile and click Next.

6.    Next, select AWS CodePipline as Configuration source and click Next.

7.    You will now see an option to add the validators to validate the configuration. Choose JSON Schema validator and add the following code:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "description": "AppConfig Validator example",
      "type": "object",
        "properties": {
    "boolEnableLimitResults": {
      "type": "boolean"
    },
    "intResultLimit": {
      "type": "number"
    }
  },
  "minProperties": 2,
    "required": [
      "intResultLimit",
      "boolEnableLimitResults"
    ]
}

8.    Next, create the configuration by clicking Create configuration profile.  Once created, you can access the CodePipeline console just by clicking on the link in the Configuration profile details box:

Step 3: Using CodeCommit as Source Repository for Application Configuration

Similar to application code, application configuration continuously changes, so it is desirable to have the same level of control on the configuration as we do on the code.

A source version control system like Git, allows multiple developers and teams to collaborate and track the change history.

We are using AWS CodeCommit as the source repository for application configuration. AWS CodeCommit is a fully-managed source control service that hosts secure Git based repositories.

1.     Navigate to AWS CodeCommit Console and create a new repository with the name ListServices.

2.    An empty repository is created. You can add the configuration file by cloning the application to your local machine and pushing the changes. However, in this scenario as it is a single configuration file.  We can do the same from the AWS Management Console by clicking Create file

3.    Use the contents from the following code and name the file as “config/configdoc.json”. The configuration has two parameters, one to enable/disable the feature in the application and second to set the limit of sample AWS services to be returned by the application.

{
  "boolEnableLimitResults": false,
  "intResultLimit": 6
}

4.    Add your name and email, and click on commit changes to add the file to the master branch as shown in below figure.

Step 4: Creating CodePipeline with AWS AppConfig as Deploy Action Provider

We will now create a CI/CD pipeline using AWS CodePipeline for releasing application configuration changes to multiple environments.

1.    Navigate to AWS CodePipeline console and create new pipeline by clicking Create pipeline.

2.    Enter the pipeline name as ListServicesConfigPipeline and choose “New service role” using the default role name.

3.    Next, we add a source stage with source provider as AWS CodeCommit and repository name as ListServices what we created in Step 3. Choose the master branch, and use Amazon CloudWatch Events to trigger our pipeline execution.

4.      Click Next to go to the build stage. As the application configuration does not require a build stage, we can skip this section.

5.    Next we add a deploy stage. Use the following values to create the deploy stage:

  • Deploy Provider: Choose AWS AppConfig
  • Region: the region you are working in.
  • Application: ListServices
  • Environment: development
  • Configuration Profile: ListServicesCodePipelineConfigProfile
  • Deployment Strategy: Choose AppConfig.AllAtOnce (optionally create a new Deployment Strategy or choose one of the pre-defined deployment Strategy – AppConfig.AllAtOnce, AppConfig.Linear50PercentEvery30Seconds and AppConfig.Canary10Percent20Minutes).
  • Artifact configuration path: config/configdoc.json

6.    Click Next to create the pipeline. It will automatically launch its first deployment and deploy the configuration to the specified environment, that is, development.

We can now take advantage of all the AWS CodePipeline features, which can be configured as required (add test actions, manual approval action, etc.). In this example, we configure AWS CodePipeline to deploy to three environments – development, testing, and production.

1.    To proceed, we will Edit the pipeline Deploy stage and add two new action groups.

2.   The values for each action group will be the same as the values used to create the previous Deploy stage except for the value environment. Use environment values as testing for one action group and production for next action group. Once added, click on save button and confirm if asked to.

 

The pipeline is now configured to pull the configuration from the CodeCommit repository and deploy to the three environments – development, testing, and production in that order.

To test the pipeline, push a change through CodeCommit console by editing the configuration file. We enable the new results limit feature by setting “boolEnableLimitResults”configuration parameter to true, and intResultLimit to any positive integer, like 5.

{
  "boolEnableLimitResults": true,
  "intResultLimit": 5
}

Once the changes are pushed to the repository, the change triggers the pipeline execution and deploys the configuration to the three environments.

Step 5: Update Lambda to fetch application configuration

Next, we will update the application code to use the application configuration from AWS AppConfig.

The Lambda function deployed through CloudFormation in step 1 is not pulling its configuration from AWS AppConfig yet.

1.    Go to CloudFormation console

2.    Open the resources tab for each CloudFormation template you launched, and click on the resource with Logical ID “ApiFunction”.  It opens the Lambda function console.

3.    Update the contents of the “Function Code” section with the following code:

const AWS = require('aws-sdk');
const appconfig = new AWS.AppConfig({ apiVersion: '2019-10-09' });
const appEnv = process.env.appEnv;

const constParams = {
  Application: 'ListServices',
  Configuration: 'ListServicesCodePipelineConfigProfile',
  Environment: appEnv
};

let cachedParams = {};
let cachedConfigData = {};
let parsedConfigData = {};
exports.handler = async (event) => {

  // Check if ClientId is defined
  if (!cachedParams.ClientId) {
    cachedParams.ClientId = create_UUID();
  }

  // Merge constParams and cachedParams 
  const params = { ...constParams, ...cachedParams };

  // Call GetConfiguration API
  const appConfigResponse = await appconfig.getConfiguration(params).promise();

  // Add ClientConfigurationVersion to cachedParams if not present
  if ((!cachedParams.ClientConfigurationVersion) || appConfigResponse.ConfigurationVersion !== cachedParams.ClientConfigurationVersion) {
    cachedParams.ClientConfigurationVersion = appConfigResponse.ConfigurationVersion;
  }

  const configData = await Buffer.from(appConfigResponse.Content, 'base64').toString('ascii');

  if ((!cachedConfigData) || (configData && cachedConfigData !== configData))
    cachedConfigData = configData;

  let result = getServices();

  if (configData == null || configData == '') {
    parsedConfigData = JSON.parse(cachedConfigData);
  } else {
    parsedConfigData = JSON.parse(configData);
  }

  if ((parsedConfigData.boolEnableLimitResults) && parsedConfigData.intResultLimit) {
    result = result.splice(0, parsedConfigData.intResultLimit);
  }

  const response = {
    statusCode: 200,
    body: JSON.stringify(result)
  };
  return response;
};

function getServices() {
  return [
    {
      name: 'AWS AppConfig'
    },
    {
      name: 'Amazon SageMaker Studio'
    },
    {
      name: 'Amazon Kendra'
    },
    {
      name: 'Amazon CodeGuru'
    },
    {
      name: 'Amazon Fraud Detector'
    },
    {
      name: 'Amazon EKS on AWS Fargate'
    },
    {
      name: 'AWS Outposts'
    },
    {
      name: 'AWS Wavelength'
    },
    {
      name: 'AWS Transit Gateway'
    },
    {
      name: 'Amazon Detective'
    }
  ];
}

function create_UUID() {
  let dt = new Date().getTime();
  const uuid = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function (c) {
    var r = (dt + Math.random() * 16) % 16 | 0;
    dt = Math.floor(dt / 16);
    return (c == 'x' ? r : (r & 0x3 | 0x8)).toString(16);
  });
  return uuid;
}

In this updated version of the Lambda function, we retrieve the application configuration from AWS AppConfig.

We do so by calling the GetConfiguration API and including the following parameters:

  • ClientId – unique identifier for this client application
  • Application – the AppConfig application we created in Step 2, that is, ListServices
  • Configuration – the AppConfig configuration profile we created in Step 2 that is, ListServicesCodePipelineConfigProfile
  • Environment – One of the three environments we created in Step 2. Please note that it is being passed from API Gateway based on the specific environment URL invoked as appEnv environment variable was defined during the stack deployment.
  • ClientConfigurationVersion – GetConfiguration API uses the value of the ClientConfigurationVersion parameter to identify the configuration version last received by your target. If you do not send the ClientConfigurationVersion parameter with each call to GetConfiguration, your target receives the most current configuration available, even if your target already has that configuration. Since you are charged based on the number of configurations received, not sending the ClientConfigurationVersion could also result in unwanted charges.

Test the application again by invoking the API operations (browsing the endpoint in the outputs tab of the CloudFormation stacks). It should only return the number of results you specified in your last commit.

Feel free to push another change to application configuration by editing the configuration file. This triggers the pipeline again. Once the deployments to are successful, you are able to see the behavior of the API changing.

Cleaning up

After you are done, clean up the resources by deleting:

  • The three CloudFormation stacks
  • The ListServicesConfigPipeline CodePipeline that we created.
  • The ListServices CodeCommit repository we created.
  • The ListServices AppConfig application we created.

Summary

In this post we saw how an independent CI/CD pipeline can be created for the application configuration using the new integration between AWS AppConfig and AWS CodePipeline.

This allows faster and seamless configuration deployments as the build stage is not required for the same. To enable automated testing of the deployment pipeline, we can also use following features of AWS AppConfig:

  • Add Validators – A validator provides a syntactic or semantic check to ensure the configuration you want to deploy functions as intended. To validate the application configuration data, we can provide a schema or a Lambda function that runs against the configuration. The configuration deployment or update can only proceed when the configuration data is valid.
  • Add Monitors –Amazon CloudWatch alarms can be configured for each environment. The system monitors alarms during a configuration deployment. If an alarm is triggered, the system rolls back the configuration

If there are any alarms, the deployment stops and will not proceed through the pipeline.

You can read more about AWS AppConfig in the User Guide.

 

About the authors

Luis Gómez is a Solutions Architect with AWS, working for public sector in Spain.  He has several years of experience building and operating cloud environments, and applying DevOps practices.  He works with customers to understand their goals and challenges, and offers prescriptive guidance to achieve their objective with AWS offerings.

 

 

 

Prasad Rao is a Partner Solutions Architect for AWS based out of UK. His focus areas are .NET Application modernization and Windows workloads on AWS. He leverages his experience to help APN Partners across EMEA for their long-term technical enablement to build scalable architecture on AWS. He also mentors diverse people who are new to cloud and would like to get started on AWS.