Declarative provisioning of AWS resources with Spinnaker and Crossplane

This post was written by Steve Borrelli, Rob Clark, Manabu McCloskey, Vikrant Kahlir, and Nima Kaviani.

In a previous blog post, we discussed how GitOps, declarative definition of infrastructure and application resources, and using technologies such as AWS Controllers for Kubernetes (ACK) and Crossplane have enabled DevOps engineers to reduce complexity and improve visibility into infrastructure rollout and application resources.

For small- and medium-sized cloud native companies, switching to a new GitOps model that takes advantage of modern GitOps tooling—such as Argo CD or Flux CD—is a relatively small barrier to entry. For bigger enterprises, however, with established CI/CD automations in place, transitioning away from their reliable mechanisms of rolling out software may not be as trivial. This is where modernization and enhancements to their existing CI/CD platforms would allow for more easily embracing newer technologies.

One of these battle-tested CI/CD platforms heavily used by some of our enterprise-grade strategic customers—such as Netflix, Autodesk, Airbnb, Salesforce, Pinterest, and Snap—is Spinnaker. In a recent blog post, we discussed the collaboration between Netflix and AWS in enabling GitOps for the Spinnaker community via enhancements done for Managed Delivery and its respective Spinnaker microservice, Keel. The intentions behind supporting managed delivery in Spinnaker are to make it easier for Spinnaker users to embrace GitOps with little to no interruption to their existing software rollout strategies. Also, to provide higher-level abstractions for defining software rollout strategies that would abstract away lower-level infrastructure complexities and allow Spinnaker users to move faster and leaner when it comes to deploying software.

In this blog post, we combine the best of these two worlds, using GitOps support in Spinnaker for declarative rollout of infrastructure and application resources via Crossplane. Crossplane is an open source project that allows teams to manage cloud resources in a Kubernetes-native way. This means you can provision and manage Amazon Simple Storage Service (Amazon S3) buckets, VPCs, databases, and Lambda functions in the same way you manage Pods, Ingresses, and StatefulSets. Because Crossplane managed resources are exposed as Kubernetes Custom Resource Definitions, a GitOps pattern can be used to manage infrastructure with Keel’s native Kubernetes support.

In the rest of this blog post, we will discuss the setup and configuration required for Spinnaker and Crossplane to work together, and steps to take for declarative deployment of infrastructure and application resources to AWS Cloud.

Prerequisites

Before getting started, ensure that you completed the following:

• Set up a Spinnaker and Keel managed delivery environment (including GitHub Webhooks).
• The AWS Command Line Interface (AWS CLI) must be installed and configured with administrative permissions.
• Configure your Spinnaker deployment to deploy to the Kubernetes cluster with Crossplane installed on it. Refer to the Spinnaker documentation or the Armory documentation for tutorials.
• The Upbound CLI must be installed.
• The kubectl Crossplane plugin must be installed.

Declarative definition of resources in Crossplane

Crossplane models cloud infrastructure as Kubernetes Custom Resource Definitions (CRDs) called managed resources. For example, in Crossplane an Amazon S3 bucket would be defined using the following YAML, which could be embedded in a Keel declarative k8s/resource:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: keel-test-bucket
  namespace: default
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: us-west-2

Like any other Custom Resource, Crossplane resources adhere to Kubernetes API conventions. The spec is validated using OpenAPI, and the forProvider stanza includes the settings that can be configured. DeletionPolicy tells Crossplane what to do when the Kubernetes resource is deleted: Delete the managed resource or orphan it.

Deploying AWS infrastructure using Keel and Crossplane

Installing Universal Crossplane with the AWS Provider

Once Spinnaker and Keel are installed, install Universal Crossplane (UXP), a CNCF-conformant distribution of Crossplane. The easiest method is using the up CLI. First, ensure that your current kubectl context is pointing to the correct cluster, our Amazon Elastic Kubernetes Service (Amazon EKS) Spinnaker cluster:

kubectl config current-context

Next, run the following command to install UXP. Refer to the documentation for configuration options and upgrading from an existing Crossplane installation.

up uxp install

To validate the installation, ensure that all the deployments are in a READY state:

kubectl get all -n upbound-system

Next, we will install a Crossplane package that includes several Compositions, including a VPC network and an Amazon EKS cluster. The package source code is available at platform-ref-aws.

A Crossplane Composition allows an infrastructure team to create an abstraction that combines a number of managed resources in an opinionated way, providing fewer things that end users could configure. For example, one could define a ContainerTestEnvironment that includes a VPC, an Amazon EKS cluster, and Amazon Simple Queue Service (Amazon SQS) queues that only allow testers to define the name of the environment and number of nodes in the cluster.

kubectl crossplane install configuration registry.upbound.io/upbound/platform-ref-aws:v0.1.4

This package will pull down the AWS and Helm Crossplane providers, and install custom compositions. To validate the package installation, run the following command to ensure that all components are installed and healthy:

kubectl get pkg

This command returns one configuration package and two providers (AWS, Helm).

NAME                                                       INSTALLED   HEALTHY   PACKAGE                                               AGE
configuration.pkg.crossplane.io/upbound-platform-ref-aws   True        True      registry.upbound.io/upbound/platform-ref-aws:v0.1.4   2d17h
 
NAME                                                  INSTALLED   HEALTHY   PACKAGE                                               AGE
provider.pkg.crossplane.io/crossplane-provider-aws    True        True      registry.upbound.io/crossplane/provider-aws:v0.18.1   2d17h
provider.pkg.crossplane.io/crossplane-provider-helm   True        True      registry.upbound.io/crossplane/provider-helm:v0.7.0   2d17h

Setting up AWS authentication to Crossplane

We also want to load our AWS credentials into the control cluster so that Crossplane is able to provision infrastructure on our behalf. We will load the credentials into a Kubernetes secret and define a Crossplane ProviderConfiguration that references the secret.

The Crossplane provider expects a secret to be in the following format:

[default]
aws_access_key_id = AK....
aws_secret_access_key = YQ+.....

Then generate a Kubernetes secret using the file:

kubectl create secret generic aws-creds -n crossp-system --from-file=creds=./creds.conf

Apply the following ProviderConfig to the Crossplane Kubernetes cluster:

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
 name: default
spec:
 credentials:
   source: Secret
   secretRef:
     namespace: upbound-system
     name: aws-creds
     key: creds

For more information about setting up AWS provider credentials, refer to the documentation.

The following resources should now exist in the Crossplane cluster:

secret/aws-creds created
provider.aws.crossplane.io/defaul created

Deploying infrastructure with Keel

The examples for creating Amazon SQS and Amazon S3 buckets can be found in the following git repository. In this example, we define an Amazon SQS queue and an Amazon S3 bucket to be created. Update your .spinnaker/spinnaker.yaml file with the following. (Be sure to replace the placeholder value for the bucket’s name.)

name: test1
application: keelguide
serviceAccount: keeldemo-service-account
artifacts: []
environments:
 - name: dev
   locations:
     account: deploy-keel
     regions: []
   resources:
   - kind: k8s/resource@v1
     spec:
       metadata:
         application: keelguide
       template:
         apiVersion: s3.aws.crossplane.io/v1beta1
         kind: Bucket
         metadata:
           name: 
         spec:
           deletionPolicy: Delete
           forProvider:
             acl: private
             locationConstraint: us-west-2
             accelerateConfiguration:
               status: Enabled
             versioningConfiguration:
               status: Enabled
             tagging:
               tagSet:
                 - key: owner
                   value: web-developers
                 - key: environment
                   value: integration
   - kind: k8s/resource@v1
     spec:
       metadata:
         application: keelguide
       template:
         apiVersion: sqs.aws.crossplane.io/v1beta1
         kind: Queue
         metadata:
           name: example-queue
         spec:
           deletionPolicy: Delete
           forProvider:
             region: us-west-2

Once the file is committed to your repository, Keel will pass the resource definition to Crossplane controllers and monitor its progress to ensure desired states are reached. Running the following commands should show the AWS resources being created:

kubectl get bucket
kubectl get queue

Verify that resources are being created through the AWS CLI:

aws --region us-west-2 s3 ls 
aws --region us-west-2 sqs list-queues

The Spinnaker UI will show these resources as being managed.

Creating a VPC via Compositions

When we installed the Crossplane Configuration package, it contained Compositions for CompositeNetwork that includes a VPC with subnets, route tables, and gateways, and a CompositeCluster that includes an Amazon EKS cluster with AWS Identity and Access Management (IAM) roles and a Node pool. This allows an infrastructure team to provide a custom resource that an end user can consume.

To deploy a CompositeNetwork in Keel, include the following in the .spinnaker/spinnaker.yml file:

name: test1
application: keelguide
serviceAccount: keeldemo-service-account
artifacts: []
environments:
 - name: dev
   locations:
     account: deploy-keel
     regions: []
   resources:
   - kind: k8s/resource@v1
     spec:
       metadata:
         application: ledemo
       template:
         apiVersion: aws.platformref.crossplane.io/v1alpha1
         kind: CompositeNetwork
         metadata:
           name: dev-network
         spec:
           id: spinnaker-dev-network
           clusterRef:
             id: spinnaker-dev-k8s

This resource creation can be verified with the following commands:

kubectl get compositenetwork dev-network 
aws ec2 describe-vpcs

The Spinnaker UI will show it as a managed resource.

Cleanup

Get all of the resources and delete them:

Kubectl get compositenetwork
Kubectl get compostitecluster

kubectl get managed

Remove Crossplane providers and configuration packages:

kubectl delete configuration.pkg.crossplane.io platform-ref-aws
kubectl delete provider.pkg.crossplane.io provider-aws
kubectl delete provider.pkg.crossplane.io provider-helm

Remove all Spinnaker microservices and its resources:

# Remove Spinnaker microservices
kustomize build spinnaker-config/overlays/keel | kubectl delete -f -
# Remove Crossplane from cluster
up uxp uninstall
Kubectl delete secret aws-creds -n upbound-system 
# Delete EKS Cluster
eksctl delete cluster -f eks/cluster.yaml

Conclusion and next steps

For companies looking to modernize their established CI/CD systems, the combination of Spinnaker’s Keel service with Upbound’s Crossplane enables the declarative provisioning of both infrastructure and AWS application resources. By architecting around the Kubernetes API, enterprises can adopt the same methodologies as large cloud providers use for their infrastructure.

This approach enables centralized infrastructure teams to publish safe, secure, and governed infrastructure APIs, making them available to developers for rapid use and self-service. Also, it is a natural evolution from infrastructure as code methodologies and tooling, such as Terraform. Enterprises operating at scale are testing the limitations of the architecture, and moving to a more cloud-native approach could be an evolution of their CI/CD tooling.

Get involved

Spinnaker Keel and Crossplane are open source projects. We would love for you to join the community and start contributing. Join the Keel community on Slack and GitHub. Join the Crossplane community on Slack and Github.