AWS Compute Blog

Managing Cross-Account Serverless Microservices

This post courtesy of Michael Edge, Sr. Cloud Architect – AWS Professional Services

Applications built using a microservices architecture typically result in a number of independent, loosely coupled microservices communicating with each other, synchronously via their APIs and asynchronously via events. These microservices are often owned by different product teams, and these teams may segregate their resources into different AWS accounts for reasons that include security, billing, and resource isolation. This can sometimes result in the following challenges:

  • Cross-account deployment: A single pipeline must deploy a microservice into multiple accounts; for example, a microservice must be deployed to environments such as DEV, QA, and PROD, all in separate accounts.
  • Cross-account lookup: During deployment, a resource deployed into one AWS account may need to refer to a resource deployed in another AWS account.
  • Cross-account communication: Microservices executing in one AWS account may need to communicate with microservices executing in another AWS account.

In this post, I look at ways to address these challenges using a sample application composed of a web application supported by two serverless microservices. The microservices are owned by different product teams and deployed into different accounts using AWS CodePipeline, AWS CloudFormation, and the Serverless Application Model (SAM). At runtime, the microservices communicate using an event-driven architecture that requires asynchronous, cross-account communication via an Amazon Simple Notification Service (Amazon SNS) topic.

Sample application

First, look at the sample application I use to demonstrate these concepts. In the following overview diagram, you can see the following:

  • The entire application consists of three main services:
    • A Booking microservice, owned by the Booking account.
    • An Airmiles microservice, owned by the Airmiles account.
    • A web application that uses the services exposed by both microservices, owned by the Web Channel account.
  • The Booking microservice creates flight bookings and publishes booking events to an SNS topic.
  • The Airmiles microservice consumes booking events from the SNS topic and uses the booking event to calculate the airmiles associated with the flight booking. It also supports querying airmiles for a specific flight booking.
  • The web application allows an end user to make flight bookings, view flights bookings, and view the airmiles associated with a flight booking.
  • In the sample application, the Booking and Airmiles microservices are implemented using AWS Lambda. Together with Amazon API Gateway, Amazon DynamoDB, and SNS, the sample application is completely serverless.
Sample microservices application

Sample Serverless microservices application

The typical booking flow would be triggered by an end user making a flight booking using the web application, which invokes the Booking microservice via its REST API. The Booking microservice persists the flight booking and publishes the booking event to an SNS topic to enable sharing of the booking with other interested consumers. In this sample application, the Airmiles microservice subscribes to the SNS topic and consumes the booking event, using the booking information to calculate the airmiles. In line with microservices best practices, both the Booking and Airmiles microservices store their information in their own DynamoDB tables, and expose an API (via API Gateway) that is used by the web application.

Setup

Before you delve into the details of the sample application, get the source code and deploy it.

Cross-account deployment of Lambda functions using CodePipeline has been previously discussed by my colleague Anuj Sharma in his post, Building a Secure Cross-Account Continuous Delivery Pipeline. This sample application builds upon the solution proposed by Anuj, using some of the same scripts and a similar account structure. To make it feasible for you to deploy the sample application, I’ve reduced the number of accounts needed down to three accounts by consolidating some of the services. In the following diagram, you can see the services used by the sample application require three accounts:

  • Tools: A central location for the continuous delivery/deployment services such as CodePipeline and AWS CodeBuild. To reduce the number of accounts required by the sample application, also deploy the AWS CodeCommit repositories here, though typically they may belong in a separate Dev account.
  • Booking: Account for the Booking microservice.
  • Airmiles: Account for the Airmiles microservice.

Without consolidation, the sample application may require up to 10 accounts: one for Tools, and three accounts each for Booking, Airmiles and Web Application (to support the DEV, QA, and PROD environments).

Account structure for sample application

Account structure for sample application

To follow the rest of this post, clone the repository in step 1 below. To deploy the application on AWS, follow steps 2 and 3:

Challenge 1: Cross-account deployment using CodePipeline

Though the Booking pipeline executes in the Tools account, it deploys the Booking Lambda functions into the Booking account.

  1. In the sample application code repository, open the ToolsAcct/code-pipeline.yaml CloudFormation template.
  2. Scroll down to the Pipeline resource and look for the DeployToTest pipeline stage (shown below). There are two AWS Identity and Access Management (IAM) service roles used in this stage that allow cross-account activity. Both of these roles exist in the Booking account:
    • Under Actions.RoleArn, find the service role assumed by CodePipeline to execute this pipeline stage in the Booking account. The role referred to by the parameter NonProdCodePipelineActionServiceRole allows access to the CodePipeline artifacts in the S3 bucket in the Tools account, and also access to the AWS KMS key needed to encrypt/decrypt the artifacts.
    • Under Actions.Configuration.RoleArn, find the service role assumed by CloudFormation when it carries out the CHANGE_SET_REPLACE action in the Booking account.

These roles are created in the CloudFormation template NonProdAccount/toolsacct-codepipeline-cloudformation-deployer.yaml.

- Name: DeployToTest
  Actions:
    - Name: CreateChangeSetTest
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Version: 1
        Provider: CloudFormation
      Configuration:
        ChangeSetName: !Join ['-',[!Ref ProjectName, 'lambda', 'CS']]
        ActionMode: CHANGE_SET_REPLACE
        StackName: !Join ['-',[!Ref ProjectName, 'lambda']]
        Capabilities: CAPABILITY_NAMED_IAM
        TemplatePath: BuildOutput::output-sam-template.yml
        TemplateConfiguration: BuildOutput::sam-config.json
        RoleArn: !Ref NonProdCloudFormationServiceRole
      InputArtifacts:
        - Name: BuildOutput
      RunOrder: 1
      RoleArn: !Ref NonProdCodePipelineActionServiceRole
    - Name: DeployChangeSetTest
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Version: 1
        Provider: CloudFormation
      Configuration:
        ChangeSetName: !Join ['-',[!Ref ProjectName, 'lambda', 'CS']]
        ActionMode: CHANGE_SET_EXECUTE
        StackName: !Join ['-',[!Ref ProjectName, 'lambda']]
      InputArtifacts:
        - Name: BuildOutput
      RunOrder: 2
      RoleArn: !Ref NonProdCodePipelineActionServiceRole

Challenge 2: Cross-account stack lookup using custom resources

Asynchronous communication is a fairly common pattern in microservices architectures. A publishing microservice publishes an event that consumers may be interested in, without any concern for who those consumers may be.

In the case of the sample application, the publisher is the Booking microservice, publishing a flight booking event onto an SNS topic that exists in the Booking account. The consumer is the Airmiles microservice in the Airmiles account. To enable the two microservices to communicate, the Airmiles microservice must look up the ARN of the Booking SNS topic at deployment time in order to set up a subscription to it.

To enable CloudFormation templates to be reused, you are not hardcoding resource names in the templates. Because you allow CloudFormation to generate a resource name for the Booking SNS topic, the Airmiles microservice CloudFormation template must look up the SNS topic name at stack creation time. It’s not possible to use cross-stack references, as these can’t be used across different accounts.

However, you can use a CloudFormation custom resource to achieve the same outcome. This is discussed in the next section. Using a custom resource to look up stack exports in another stack does not create a dependency between the two stacks, unlike cross-stack references. A stack dependency would prevent one stack being deleted if another stack depended upon it. For more information, see Fn::ImportValue.

Using a Lambda function as a custom resource to look up the booking SNS topic

The sample application uses a Lambda function as a custom resource. This approach is discussed in AWS Lambda-backed Custom Resources.
Walk through the sample application and see how the custom resource is used to list the stack export variables in another account and return these values to the calling AWS CloudFormation stack. Examine each of the following aspects of the custom resource:

  1. Deploying the custom resource.
  2. Custom resource IAM role.
  3. A calling CloudFormation stack uses the custom resource.
  4. The custom resource assumes a role in another account.
  5. The custom resource obtains the stack exports from all stacks in the other account.
  6. The custom resource returns the stack exports to the calling CloudFormation stack.
  7. The custom resource handles all the event types sent by the calling CloudFormation stack.

Step1: Deploying the custom resource

The custom Lambda function is deployed using SAM, as are the Lambda functions for the Booking and Airmiles microservices. See the CustomLookupExports resource in Custom/custom-lookup-exports.yml.

Step2: Custom resource IAM role

The CustomLookupExports resource in Custom/custom-lookup-exports.yml executes using the CustomLookupLambdaRole IAM role. This role allows the custom Lambda function to assume a cross account role that is created along with the other cross account roles in the NonProdAccount/toolsacct-codepipeline-cloudformation-deployer.yml. See the resource CustomCrossAccountServiceRole.

Step3: The CloudFormation stack uses the custom resource

The Airmiles microservice is created by the Airmiles CloudFormation template Airmiles/sam-airmile.yml, a SAM template that uses the custom Lambda resource to look up the ARN of the Booking SNS topic. The custom resource is specified by the CUSTOMLOOKUP resource, and the PostAirmileFunction resource uses the custom resource to look up the ARN of the SNS topic and create a subscription to it.

Because the Airmiles Lambda function is going to subscribe to an SNS topic in another account, it must grant the SNS topic in the Booking account permissions to invoke the Airmiles Lambda function in the Airmiles account whenever a new event is published. Permissions are granted by the LambdaResourcePolicy resource.

Step4: The custom resource assumes a role in another account

When the Lambda custom resource is invoked by the Airmiles CloudFormation template, it must assume a role in the Booking account (see Step 2) in order to query the stack exports in that account.

This can be seen in Custom/custom-lookup-exports.py, where the AWS Simple Token Service (AWS STS) is used to obtain a temporary access key to allow access to resources in the account referred to by the environment variable: ‘CUSTOM_CROSS_ACCOUNT_ROLE_ARN’. This environment variable is defined in the Custom/custom-lookup-exports.yml CloudFormation template, and refers to the role created in Step 2.

sts = boto3.client('sts')
assumedRole = sts.assume_role(
    RoleArn=os.environ['CUSTOM_CROSS_ACCOUNT_ROLE_ARN'],
    RoleSessionName='LambdaCloudFormationSession'
)
credentials = assumedRole['Credentials']
accessKey = credentials['AccessKeyId']
secretAccessKey = credentials['SecretAccessKey']
sessionToken = credentials['SessionToken']
cfn = boto3.client('cloudformation',  
	region_name=os.environ['AWS_DEFAULT_REGION'], 
aws_access_key_id=accessKey,
aws_secret_access_key=secretAccessKey, 
	aws_session_token=sessionToken)

Step5: The custom resource obtains the stack exports from all stacks in the other account

The function get_exports() in Custom/custom-lookup-exports.py uses the AWS SDK to list all the stack exports in the Booking account.

Step 6: The custom resource returns the stack exports to the calling CloudFormation stack

The calling CloudFormation template, Airmiles/sam-airmile.yml, uses the custom resource to look up a stack export with the name of booking-lambda-BookingTopicArn. This is exported by the CloudFormation template Booking/sam-booking.yml.

PostAirmileFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: post-airmiles-lambda.handler
    Runtime: python2.7
    Policies: AmazonDynamoDBFullAccess
    Events:
      SubmitBooking:
        Type: SNS
        Properties:
          Topic:
            !GetAtt CUSTOMLOOKUP.booking-lambda-BookingTopicArn

Step7: The custom resource handles all the event types sent by the calling CloudFormation stack

Custom resources used in a stack are called whenever the stack is created, updated, or deleted. They must therefore handle CREATE, UPDATE, and DELETE stack events. If your custom resource fails to send a SUCCESS or FAILED notification for each of these stack events, your stack may become stuck in a state such as CREATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS. To handle this cleanly, use the crhelper custom resource helper. This strongly encourages you to handle the CREATE, UPDATE, and DELETE CloudFormation stack events.

Challenge 3: Cross-account SNS subscription

The SNS topic is specified as a resource in the Booking/sam-booking.yml CloudFormation template. To allow an event published to this topic to trigger the Airmiles Lambda function, permissions must be granted on both the Booking SNS topic and the Airmiles Lambda function. This is discussed in the next section.

Permissions on booking SNS topic

SNS topics support resource-based policies, which allow a policy to be attached directly to a resource specifying who can access the resource. This policy can specify which accounts can access a resource, and what actions those accounts are allowed to perform. This is the same approach as used by a small number of AWS services, such as Amazon S3, KMS, Amazon SQS, and Lambda. In Booking/sam-booking.yml, the SNS topic policy allows resources in the Airmiles account (referenced by the parameter NonProdAccount in the following snippet) to subscribe to the Booking SNS topic:

BookingTopicPolicy:
  Type: AWS::SNS::TopicPolicy
  Properties:
    PolicyDocument:
      Id: BookingTopicPolicy
      Version: '2012-10-17'
      Statement:
      - Sid: BookingTopicPolicy-stmt1
        Effect: Allow
        Principal:
            AWS:
              - !Sub arn:aws:iam::${AWS::AccountId}:root
        Action:
          - sns:Publish
          - sns:RemovePermission
          - sns:SetTopicAttributes
          - sns:DeleteTopic
          - sns:ListSubscriptionsByTopic
          - sns:GetTopicAttributes
          - sns:Receive
          - sns:AddPermission
          - sns:Subscribe
        Resource: !Ref BookingTopic
      - Sid: BookingTopicPolicy-stmt2
        Effect: Allow
        Principal:
            AWS:
              - !Sub arn:aws:iam::${NonProdAccount}:root
        Action:
          - sns:Subscribe
          - sns:ListSubscriptionsByTopic
        Resource: !Ref BookingTopic
    Topics:
    - !Ref BookingTopic

Permissions on the Airmiles Lambda function

The Airmiles microservice is created by the Airmiles/sam-airmile.yml CloudFormation template. The template uses SAM to specify the Lambda functions together with their associated API Gateway configurations. SAM hides much of the complexity of deploying Lambda functions from you.

For instance, by adding an ‘Events’ resource to the PostAirmileFunction in the CloudFormation template, the Airmiles Lambda function is triggered by an event published to the Booking SNS topic. SAM creates the SNS subscription, as well as the permissions necessary for the SNS subscription to trigger the Lambda function.

PostAirmileFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: post-airmiles-lambda.handler
    Runtime: python2.7
    Policies: AmazonDynamoDBFullAccess
    Events:
      SubmitBooking:
        Type: SNS
        Properties:
          Topic:
            !GetAtt CUSTOMLOOKUP.booking-lambda-BookingTopicArn
    Environment:
      Variables: 
        TABLE_NAME: !Ref AirmileDBTable

However, the Lambda permissions generated automatically by SAM are not sufficient for cross-account SNS subscription, which means you must specify an additional Lambda permissions resource in the CloudFormation template. LambdaResourcePolicy, in the following snippet, specifies that the Booking SNS topic is allowed to invoke the Lambda function PostAirmileFunction in the Airmiles account.

LambdaResourcePolicy:
  Type: AWS::Lambda::Permission
  Properties:
    FunctionName: !GetAtt PostAirmileFunction.Arn
    Principal: sns.amazonaws.com
    Action: lambda:InvokeFunction
    SourceArn : !GetAtt CUSTOMLOOKUP.booking-lambda-BookingTopicArn

Summary

In this post, I showed you how product teams can use CodePipeline to deploy microservices into different AWS accounts and different environments such as DEV, QA, and PROD. I also showed how, at runtime, microservices in different accounts can communicate securely with each other using an asynchronous, event-driven architecture. This allows product teams to maintain their own AWS accounts for billing and security purposes, while still supporting communication with microservices in other accounts owned by other product teams.

Acknowledgments

Thanks are due to the following people for their help in preparing this post:

  • Kevin Yung for the great web application used in the sample application
  • Jay McConnell for crhelper, the custom resource helper used with the CloudFormation custom resources