Listing Serverless Applications with Your Amazon Machine Images on AWS Marketplace

By James Loope, Solutions Architect at AWS
By KC Li, Product Manager at AWS

Independent software vendors (ISVs) are using serverless technologies to automate tasks, process data, build integrations, and auto scale their solutions.

By using serverless applications alongside existing Amazon Machine Images (AMIs), ISVs can provide customers with solutions that are easier to use, more elastic, and more scalable.

AWS Marketplace recently announced a new feature that enables sellers to publish solutions comprised of AMIs and serverless applications that customers can deploy with a few simple clicks using AWS CloudFormation.

This makes it easier for customers to deploy these software solutions, rather than ISVs packaging AWS Lambda code into an AMI or referencing their own Amazon Simple Storage Service (Amazon S3) bucket.

Customers can more easily review the vendors’ architecture and deploy without granting additional permissions for the AMI to provision Lambda functions or downloading code from the ISV’s S3 bucket.

Customers can always find the latest versions of the AMIs and associated serverless applications in 16 supported AWS regions. With this approach, it’s also easier for sellers to update their serverless applications separately from their AMIs.

In this post, we will explain how ISVs can offer AMI-based solutions with serverless application components using CloudFormation and the Serverless Application Model (SAM). We’ll also discuss the benefits of breaking apart the serverless application from the AMI compared with including the Lambda code inline.

Figure 1 below shows the account and security topology of the Marketplace SAM publishing flow.

Figure 1 – AWS Marketplace serverless publishing.

In Part 1 of the post, we’ll walk through an example of how to publish a Hello World Lambda to the Serverless Application Repository using sam-cli, and demonstrate deployment through AWS Marketplace using CloudFormation.

Learn more about using CloudFormation on AWS Marketplace.

If you have an existing Lambda function defined in CloudFormation and would like to take advantage of this feature, Part 2 of this post showcases XebiaLabs’ experience re-factoring Lambda resources to use AWS Serverless Application Repository.

Part 1: Publishing a Lambda to Serverless Application Repository

In order to demonstrate the AWS Marketplace serverless application functionality, we’ll need a simple Lambda function to deploy.

We’ll use a simple HelloWorld in JavaScript using the Serverless Application Model, an extension of AWS CloudFormation that provides simplified syntax to define serverless resources, including Lambda functions, Amazon API Gateway APIs, and Amazon DynamoDB tables.

We’ll package and publish this application to the Serverless Application Repository, which is a managed repository for serverless applications that enables teams, organizations, and individual developers to store and share reusable applications so they can easily assemble and deploy serverless architectures.

HelloWorld Lambda (index.js):

exports.handler = async function(event,context) {
return 'Hello world';
};

Ensure that aws-cli and aws-sam-cli are installed and updated to the latest version. Older versions may not have complete support for this example.

Example upgrade command using pip:

pip install awscli —upgrade
pip install aws-sam-cli --upgrade

Next, we will model our Lambda serverless function in CloudFormation using the SAM syntax and specifying the “AWS::Serverless-2016-10-31” transform.

In order for the serverless repo to publish our application, the “AWS::ServerlessRepo::Application” metadata key must be included. We’ve filled the required fields with sample data in the example below.

SAM application template (template.yml):

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs8.10

Metadata:
  AWS::ServerlessRepo::Application:
    Name: HelloWorld
    Description: hello world
    Author: user1
    SpdxLicenseId: Apache-2.0
    LicenseUrl: LICENSE.txt
    ReadmeUrl: README.md
    Labels: ['tests']
    HomePageUrl: https://github.com/user1/my-app-project
    SemanticVersion: 0.0.1
    SourceCodeUrl: https://github.com/user1/my-app-project

When the aws-sam-cli packages our Lambda, it needs an Amazon S3 bucket to store the assets. This bucket is used to store your Lambda and share it with the AWS Serverless Application Repository. You can use an existing bucket, or create a new one as we have here.

aws s3 mb s3://jloope-sam-mp-test --region us-east-1

After we’ve packaged our Lambda assets and pushed them to Amazon S3, we’ll need to allow the serverless repository access to our S3 bucket and packaged assets. To do this, we have created a policy document allowing the “serverlessrepo” principal access to the GetObject method on the S3 bucket that we’ve used to package the Lambda.

Insert your bucket name into the policy document below (serverlessrepo-bucket-policy.json):

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "serverlessrepo.amazonaws.com"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::your-bucket-here/*"
}
]
}

Run the following command, using your own S3 bucket name, to create the bucket policy allowing the serverless repo access to your bucket:

aws s3api put-bucket-policy --bucket your-bucket-here --policy file://serverlessrepo-bucket-policy.json

Now that we have a Lambda, the SAM template describing it, a bucket to package it, and the proper policy attached, we can push our application to the Serverless Application Repository.

Next, we’ll package the SAM application which will push the Lambda code package to our S3 bucket and write a new SAM template with the appropriate CodeUri describing its S3 location. In the following example, the new template is saved as “packaged.yml”.

sam package --template-file template.yml --output-template-file packaged.yml --s3-bucket your-bucket-here --region us-east-1

And then we’ll publish the packaged SAM application using the output file of the package operation, “packaged.yml”.

sam publish --template packaged.yml --region us-east-1

Receiving the following advice upon success:

Publish Succeeded
Created new application with the following metadata:
{
  "SpdxLicenseId": "Apache-2.0", 
  "Name": "HelloWorld", 
  "Author": "user1", 
  "Labels": [
    "tests"
  ], 
  "SourceCodeUrl": "https://github.com/user1/my-app-project", 
  "LicenseUrl": "s3://your-bucket-here/d41d8cd98f00b204e9800998ecf8427e", 
  "SemanticVersion": "0.0.1", 
  "ReadmeUrl": "s3://your-bucket-here/d41d8cd98f00b204e9800998ecf8427e", 
  "HomePageUrl": "https://github.com/user1/my-app-project", 
  "Description": "hello world"
}
Click the link below to view your application in AWS console:
https://console.aws.amazon.com/serverlessrepo/home?region=us-east-1#/published-applications/arn:aws:serverlessrepo:us-east-1:XXXXXXXXXXXX:applications~HelloWorld

Click the link returned from completion of the publish command and collect the Id/ARN of your SAM application from the AWS Console.

Figure 2 – Serverless application details page in AWS Console.

Next, we’ll share our SAM application with AWS Marketplace by creating an application policy using the application ARN collected above. This policy allows the AWS Marketplace ingestion principal to perform the deploy action of the application on behalf of your entitled customer.

aws serverlessrepo put-application-policy \
--application-id arn:aws:serverlessrepo:us-east-1:800644653141:applications/HelloWorld \
--statements Principals=assets.marketplace.amazonaws.com,Actions=Deploy \
--region us-east-1

Finally, we declare our published SAM application in the product CloudFormation template specifying the ApplicationID (ARN) that we’ve published and shared in the above steps with the accompanying SemanticVersion.

Transform: AWS::Serverless-2016-10-31
Description: An example of Master CFT template with SAR application

Transform: AWS::Serverless-2016-10-31
Description: An example of Master CFT template with SAR application

Resources:
  SampleSARApplication:
    Type: AWS::Serverless::Application
    Properties:
      Location:
         ApplicationId: arn:aws:serverlessrepo:us-east-1:800644653141:applications/HelloWorld
         SemanticVersion: 0.0.1
  SampleEC2Instance:
    Type: AWS::EC2::Instance
      Properties: 
        ImageId: "ami-79fd7eee"
        KeyName: "testkey"
        BlockDeviceMappings: 
          - DeviceName: "/dev/sdm"
            Ebs: 
              VolumeType: "io1"
              Iops: "200"
              DeleteOnTermination: "false"
              VolumeSize: "20"
          - DeviceName: "/dev/sdk"
            NoDevice: {}

The remainder of your product CloudFormation—including the AMI instantiation and any other resources, policies, and roles—can be implemented as you would otherwise, keeping with existing AWS Marketplace policies and best practices for CloudFormation listings.

Learn more about CloudFormation listings on AWS Marketplace.

In the next part of this post, we’ll examine an example from XebiaLabs where we’ll refactor an existing inline Lambda and replace it with a newly migrated SAM application.

Part 2: Refactoring from Lambda to Serverless

AWS Marketplace now supports Lambda assets with AWS Serverless Application Model. If you have existing Lambda assets deployed via CloudFormation, some re-factoring will be necessary to take advantage of this feature.

In order to re-factor an existing Lambda defined in CloudFormation, we must move our Lambda code into a new SAM project and create a new SAM application template.

In the existing master template, the Lambda resource must be replaced with the new Serverless::Application resource. The process for publishing your Lambda to AWS Serverless Application Repository and sharing with AWS Marketplace is described above in Part 1.

The following section will focus on the process of modifying your existing Lambda declarations to work with AWS Serverless Application Model.

This example from XebiaLabs, an AWS Partner Network (APN) Advanced Technology Partner, shows a snippet from a CloudFormation template with the original in-line Lambda. We’ve folded the code for brevity.

Resources:
  CreateRDSDatabase:
    Type: AWS::Lambda::Function
    Properties:
      Description: Lambda function to create an RDS database
      Handler: createdb.handler
      MemorySize: 128
      Role: !GetAtt RDSLambdaRole.Arn
      Runtime: python3.6
      Timeout: 60
      VpcConfig:
        SecurityGroupIds:
          - !Ref AuroraRDSSecurityGroupID
        SubnetIds: !Ref PrivateSubnetIDs
      Code: +--------------------------FOLD-------------------------------
  RDSLambdaRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: lambda_policy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - cloudformation:DescribeStacks
                Resource: '*'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole      
Outputs:
  CreateDbLambdaArn:
    Description: The ARN of the created Lambda function
    Value: !GetAtt CreateRDSDatabase.Arn

In the Lambda function to be migrated (CreateRDSDatabase), each property that has a referential value must be parametrized in the master CloudFormation template, and passed as a valid type to the child template in the serverless::application::properties::parameters stanza.

Dependent resources that have no relationships to resources outside of the serverless application, such as the RDSLambdaRole in this example, can be easily moved from the master template into the application template. Finally, any referential dependency on objects inside the new serverless application need to be exposed through an output key.

In the next code block, you can see the Lambda function re-factored into a serverless function. Below, we’ll discuss each of the necessary modifications.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: |
  An app that creates databases into a RDS instance
Globals:
  Function:
    Timeout: 60
Parameters:
  AuroraRDSSecurityGroupID:
    Type: AWS::EC2::SecurityGroup::Id
  PrivateSubnetIDs:
    Type: List<AWS::EC2::Subnet::Id>
Resources:
  CreateRDSDatabaseFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri:
        Bucket: <%REPO_BUCKET%>
        Key: b62cdab3-009d-473d-8860-662e6f855a40
      Handler: createdb.handler
      MemorySize: 128
      Role:
        Fn::GetAtt:
        - RDSLambdaRole
        - Arn
      Runtime: python3.7
      VpcConfig:
        SecurityGroupIds:
        - Ref: AuroraRDSSecurityGroupID
        SubnetIds:
          Ref: PrivateSubnetIDs
  RDSLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
      Path: /
      Policies:
      - PolicyDocument:
          Statement:
          - Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Effect: Allow
            Resource: arn:aws:logs:*:*:*
          - Action:
            - cloudformation:DescribeStacks
            Effect: Allow
            Resource: '*'
          Version: '2012-10-17'
        PolicyName: lambda_policy                
Outputs:
  CreateRDSDatabaseFunction:
    Description: Create RDS Database Function Lambda Function ARN
    Value:
      Fn::GetAtt:
      - CreateRDSDatabaseFunction
      - Arn

First, you’ll see that we’ve added ‘Transform: AWS::Serverless-2016-10-31’ to the top of the template. This is necessary for any template that uses AWS Serverless Application Model resources. It also needs to be added to the master CloudFormation template.

Second, the Lambda code location has been modified to refer to assets now being published from S3 to the Serverless Application Repository. The process for publishing your Lambda to AWS Serverless Application Repository and sharing with AWS Marketplace is described in Part 1.

Next, the SecurityGroupIds and SubnetIds keys in the VpcConfig property of the Lambda have been added to the parameters stanza. They are then referenced from within the Serverless::Function resource CreateRDSDatabaseFunction. Any other properties necessary to the function of your Lambda can also be passed in this way.

Finally, the RdsLambdaRole resource has been moved from the master template into the SAM application template. The resource name (ARN) of the CreateRDSDatabaseFunction is passed into an output key for use in the master template.

Other values from resources created in the SAM application can also be returned from the serverless application to the master template in the same way.

The next example describes an excerpt of the CloudFormation template after we have re-factored from using AWS::Lambda::Function to the AWS::Serverless:Application.

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: >
  Template that creates a Lambda function that can create databases in a provisioned Aurora PostgreSQL cluster.
Parameters: 
  AuroraRDSSecurityGroupID:
    Description: Aurora Security Group
    Type: 'AWS::EC2::SecurityGroup::Id'
  PrivateSubnetIDs:
    Description: The subnets the Lambda should be deployed to
    Type: List<AWS::EC2::Subnet::Id>
Resources:
    CreateDBApplication:
      Type: AWS::Serverless::Application
      Properties:
        Location:
          ApplicationId: arn:aws:serverlessrepo:us-east-1:XXXXX:applications/create-databases-in-rds
          SemanticVersion: 0.0.3
        Parameters:
          AuroraRDSSecurityGroupID: !Ref AuroraRDSSecurityGroupID
          PrivateSubnetIDs: !Join [ ",", !Ref PrivateSubnetIDs ]
      RDSCreateDatabases:
        Type: Custom::RDSCreateDatabases
        Properties:
            ServiceToken: !GetAtt
            - CreateDBApplication
            - Outputs.CreateRDSDatabaseFunction

The new statement references the ApplicationId and SemanticVersion we obtained from publishing our application template and code to AWS Serverless Application Repository.

The VPCConfig properties AuroraRDSSecurityGroupID and PrivateSubnetIDs are now passed as parameters, as we have moved the properties themselves into the serverless application template.

Lastly, the Custom:: type at the end references the CreateRDSDatabaseFunction output from the serverless application in order to consume the ARN of the Lambda created within the serverless application.

Summary

In this post, we introduced you to a new option for ISVs to publish AMI-based solutions with serverless components.

We demonstrated an example from APN Partner XebiaLabs of how you can use AWS SAM and CloudFormation to help integrate your Lambdas and accelerate your go-to-market (GTM) strategy with AWS Marketplace.

To summarize, we published our Lambda to AWS Serverless Application Repository as described in Part 1.

We added the line including the serverless transform to our templates. We parametrized each property that depends on external values and moved any uncomplicated dependent resources into the serverless application template. Lastly, we exposed any internal values needed by the master template in the outputs stanza of our application.

We hope you’ve enjoyed this simple example of how you can use AWS SAM and CloudFormation to integrate your Land accelerate your GTM strategy with AWS Marketplace.