Streamline Platform Engineering using AWS CodeStar Connections with AWS Service Catalog

Introduction

AWS Service Catalog and AWS CloudFormation now support Git-sync capabilities to allow Platform Engineers to streamline their DevOps processes by keeping their Infrastructure as Code (IaC) templates in their source control libraries like GitHub and BitBucket. These enhancements help Platform Engineers to more effectively create, version, and manage their Well-Architected patterns with application teams to accelerate development. This blog post will focus on incorporating Git-sync capabilities with Service Catalog, and you can read more about CloudFormation’s git-sync support.

Since 2016, AWS suggests that customers form a Cloud Center of Excellence (CCoE) in their organization to act as a central hub for cloud adoption, enablement, product development, and business. Mark Schwartz, AWS Enterprise Strategist, provides an update on Designing CCoEs, how they have evolved over time, and how we see customers utilize them to drive organizational change.

A critical component of the CCoE is Platform Engineering. Platform Engineering acts as the central product development and enablement team, creating frameworks to deliver working solutions to application teams and business units quickly. The Platform Engineering team is responsible for delivering reusable “Well-Architected” cloud products and “paved roads” which wrap a company’s standards for security, reliability, observability, and other common components into codified best practice patterns. In Service Catalog, these patterns are represented as IaC Terraform modules or CloudFormation templates. CCoE teams distribute these products across multi-account environments, enabling application teams to self-service and launch resources as needed.

For Platform Engineers, managing the process for creating and updating approved products can be cumbersome, but with git-sync functionality for Service Catalog, a day in the life of a Platform Engineer just got easier. In this blog post, we will dive into the code repository strategies, processes, and tools that are managed by the CCoE, allowing application teams to accelerate development without tickets, handoffs, and waiting.

Git-sync for Service Catalog fits naturally within a software development lifecycle. Platform Engineers can now update their Service Catalog products by simply committing a new template version to their code repository. DevSecOps workflows can automatically execute on the newly pushed code, scanning for code quality and best practices. Application teams can then use their favorite Infrastructure-as-code tools to consume products and customize resources to their requirements, operating securely within the bounds of controls like Service Control Policies, AWS Config Rules, AWS Control Tower Proactive Controls, and Identity and Access Management (IAM) Policies. Using this native support for DevSecOps CI/CD pipelines, application teams can feel confident that the infrastructure-as-code resources are in sync with updates from the source control repository and comply with the company’s policies.

Code Repo Strategy

A common approach when implementing version control across a large enterprise is to leverage organizations and teams within GitHub enterprise to organize repositories and access. GitHub organizations serve as a container for shared work which allows collaboration across many projects within an enterprise. A large enterprise may have multiple organizations, but in general, fewer organizations are better to enable more visibility into existing code and minimize permission management. Within a GitHub organization, you can further divide repository structure into teams (and nested teams) that reflect the teams and projects in your enterprise.

This repository strategy aligns well with the recommended hub-and-spoke pattern using Service Catalog where there is a central “hub” AWS Account that contains all Portfolios available to application teams to consume in “spoke” accounts. By specifying the Platform Engineering team (or a subset of the team) as code owners in the repositories defining Service Catalog products, this group is responsible for reviewing and approving any new Service Catalog products or changes to existing products.

Connecting Repositories with Service Catalog Products

Service Catalog Products are organized into Portfolios that application teams can use to provision infrastructure. We recommend you start with four Service Catalog Portfolios: two for individual Products that can be consumed, launched, and chained together, and two Portfolios with Composed Solutions consisting of many Products wired together. Some good examples of Composed Solutions include a Serverless Stack (Amazon API Gateway + AWS Lambda + AWS DynamoDB), or a Containers Stack (AWS Application Load Balancer + Amazon Elastic Container Service [ECS] + a relational or nonrelational database option), or a three-tier web app (Application Load Balancer [ALB] + Elastic Compute Cloud [EC2] and Relational Database Service [RDS] instance).

For each type of Portfolio, consider a production-ready Portfolio and a pilot-ready Portfolio which are identified with tags and versioning. A recommended best practice is to use semantic versioning of Service Catalog Products and Portfolios as a way to indicate to application teams which Portfolios are ready for production and which are still being tested. The Platform Engineering team can guarantee version compatibility for Composed Solutions, but not for individual Products. Teams who are willing to validate and test Products can provision Products in the pilot-ready Portfolio as part of an internal beta testing group. The promotion of Products to a production-ready Portfolio occurs after this internal validation cycle.

This chart illustrates the four types of recommended Portfolios:

The git-sync capability uses AWS CodeStar Connections to authorize the connection between your AWS account and an external repository provider. After ensuring you have the appropriate permissions on your IAM role, you will create and authorize a one-time AWS CodeStar Connection, which is part of the developer tools suite of services. You can do this programmatically using the AWS CLI, AWS CodeStar Connection APIs, or through the AWS console.

Refer to the next steps to configure an AWS CodeStar Connection in the console.

Access AWS CodeStar Connections from the Administration section in the Service Catalog console. Create the connection by selecting Create connection in the upper right corner.

Choose your provider (in this case we’re connecting to GitHub), enter a name for your connection, and select Connect to GitHub.

In the next step, select Install a new app.

The AWS Connector for GitHub installation page in GitHub will open. Choose which repositories you want to allow the git-sync connector access to, and select Install.

Congratulations! You have created the connection between AWS and your GitHub repository!

The next step is to create the product to be shared to your application teams through Service Catalog. Let’s create an opinionated VPC product that meets your company’s security, resiliency, and observability requirements. We’ll define this in CloudFormation and store the template in GitHub. Here’s a diagram of one example:

This sample opinionated VPC product is deployed into an application team’s “spoke” AWS account. Your VPC product may be different, but our example here is configured like this:

• Based in us-east-1 (North Virginia), your company’s selected AWS region
• The VPC is built using a CIDR from AWS IPAM, an IP management solution containing your company’s IP space and spread across 3x Availability Zones (AZs) to meet your company’s resiliency requirements
• The VPC contains one set of public subnets and two sets of private subnets. The public subnets allow for decentralized internet ingress and the attached Transit Gateway (TGW) provide a path for centralized internet egress
• We have observability enabled using VPC Flow Logs to track packet metadata for your company’s security requirements, AWS Cloudtrail is enabled for auditing, and Amazon CloudWatch is used for other statistics
• We also see our Service Catalog products that have been shared in a Portfolio

Why should you create a VPC Product like this? Consider the amount of time saved by application teams now that they don’t have to think about what a VPC looks like. They just get one assigned to them as part of a Service Catalog product, complete and built to the exact specifications of Platform Engineering’s security, resiliency, performance, and observability standards. This allows application teams to build faster by removing undifferentiated heavy lifting.

Let’s add this opinionated VPC Product to Service Catalog using our new GitHub Connector. From the Service Catalog console under Admin, click Product List in the left sidebar, and then Create Product.

Select the option Specify your code repository using a CodeStar provider. The repositories you successfully connected to will be available in the first drop down menu.

A common way to manage the ongoing development of Service Catalog Products is to set the main branch of the repository as the one that is synchronized with product definitions. This allows the teams to follow a branching strategy which allows for experimentation in non-main branches. With appropriate checks and reviews in place this ensures that the Products that are available in the Service Catalog are ready for use by application teams. Additional best practices include specifying the owner and contact information for the Product and to add tags to track ownership and spend via a chargeback model.

After entering the details, click Create Product. Congratulations! You’ve created a new product and linked it to your code repository!

Integrating DevSecOps

Once the git-sync connector has been enabled, you can integrate Service Catalog products into a continuous integration process using GitHub Actions so that the products available to application teams are compliant and ready to be used according to an organization’s internal processes and policies.

This flowchart shows a high-level overview of the process to review and promote a product to Service Catalog. Note that by implementing the concept of code owners for repositories managing Service Catalog Products, inner-source contributions are enabled by allowing any team to contribute suggested changes, and those changes will be reviewed by the Platform Engineering team.

Once Products are published to Portfolios in Service Catalog, customers typically share them to “spoke” AWS accounts owned by application teams. For landing zones using AWS IAM Identity Center (formally AWS SSO), this is a great opportunity to streamline sharing automation by using Service Catalog’s wildcard IAM principal name support.

Clean up

Products in the Service Catalog can be deleted in the console or via AWS CLI. It is important to note that after deleting the Product from Service Catalog it cannot be recovered. Any instances of the Product that were deployed to a spoke account will not be deleted.

Removing the connection from AWS to GitHub is possible from the AWS console under Developer Tools as shown below. Expand the Setting menu in the left sidebar to see the Connections. Select the connection you want to delete and choose the delete button.

Another option is to delete via the AWS CLI with the following command:

aws codestar-connections delete-connection --connection-arn arn:aws:codestar-connections:us-west-2:account_id:connection/aEXAMPLE-8aad-4d5d-8878-dfcab0bc441f

Next Steps

Using git-sync for Service Catalog, CCoE Platform Engineers streamline their product and pattern development with their code repositories, integrating code quality scans and best practices. This allows application teams to use their favorite Infrastructure-as-code solutions (Terraform, CloudFormation, CDK) to reference Service Catalog product IDs, input their own parameters as code, and wire together infrastructure for their applications. As a result, the software development process accelerates and engineering productivity increases across the organization with this automated integration of GitHub connector and Service Catalog. Platform Engineers and Application teams’ lives just got easier!