AWS Marketplace

Streamlining Third-party add-on management in Amazon EKS cluster using Terraform and Amazon EKS add-on catalog

Customers using Amazon Elastic Kubernetes Service (Amazon EKS) want to install and manage operational tools for making the cluster production ready. You can choose from a curated list of operational software from the Amazon EKS add-on catalog, which contains both Amazon EKS’s native add-ons and third-party add-ons from AWS Marketplace. Lifecycle of these add-ons can be done using different methods. It can be managed using actions in User Interface on the Amazon Web Services (AWS) console, Amazon Web Services Command Line Interface (AWS CLI) commands (for example, list-addons), and Amazon EKS Application Program Interface (for example, list-addons API).

In this blog post, you can learn how to use Terraform, a popular infrastructure as code (IaC) tool, for creating and managing the lifecycle of add-ons in an EKS cluster. In this post, I will show you how to find, install, and delete Amazon EKS third-party add-ons using Terraform.

Overview

At AWS re:invent 2022, we introduced an integrated deployment option for deploying Helm-based third-party products from AWS Marketplace into an existing EKS cluster as an add-on. In a previous post, Deploy third-party software add-ons from AWS Marketplace to Amazon EKS clusters, we explained on how to deploy these third-party add-ons from AWS Marketplace using the Amazon EKS console or AWS CLI. In this post, I want to show you how the same can be accomplished via Terraform. At the time of publication of this blog post, these AWS Marketplace add-ons are available for deployment into an existing Amazon EKS cluster. Down below, you will see instructions on how to describe on metadata for these add-ons, which has information available versions, supported Kubernetes version and  supported architectures.

  • Kubecost
  • Kasten
  • Datree
  • HA Proxy
  • Dynatrace
  • Kpow
  • Teleport
  • Tetrate
  • Upbound Universal Crossplane

This list grows as we add more software from independent software vendors on AWS Marketplace. For this post, I will show you how to install and manage Kasten’s K10 from the Amazon EKS add-on catalog using Terraform. Kasten K10  is a data protection platform purpose-built for Kubernetes, which provides enterprise operations teams an easy-to-use, scalable, and secure system for backup and restore, disaster recovery, ransomware defense and mobility of Kubernetes applications. Kasten K10 provides a policy-driven, extensible architecture to manage data with application auto discovery, database integrations, and a web UI for data management consistency on your EKS clusters.

Architecture diagram showing steps involved in adding third party add-on onto an Amazon EKS cluster such as discover, subscribe and deploy

ALT TXT: In the architecture diagram above, you can see an Amazon EKS User discover Kasten K10 addon from the Amazon EKS Addon catalog, subscribe to the product in aws marketplace and deploy the addon into the EKS cluster.

Prerequisites

This blog assumes you are familiar with how to set up EKS cluster with Terraform, if not kindly follow the steps here first to understand how to set up Terraform modules for setting up an EKS cluster.

Solution walkthrough

For managing Amazon EKS third-party add-ons via Terraform, you need to define the add-on under the existing EKS cluster’s Terraform module or as a separate module. To start with the declaration of add-on, you need to know the third-party add-on’s name and any additional configurations (if required). You can use the Amazon EKS APIs to fetch these details. This blog post expands on Terraform’s EKS add-on module to showcase the installation and management of Marketplace add-ons in an EKS cluster.

Step 1

To know the version information of the add-on, you can use the available AWS CLI for Amazon EKS commands to describe the add-on. This task is a one-time process. Kasten K10 IAM Policy that needs to be performed for each new add-on.

Some third-party add-ons might need custom configurations. Use the `describe-addon-configuration` AWS CLI command to describe the JSON schema of the configuration. Here is the command to describe kubecost add-on versions and their configurations.

aws eks describe-addon-versions --addon-name kubecost_kubecost
        
aws eks describe-addon-configuration --addon-name kubecost_kubecost --addon-version v1.10.1-eksbuild.1

Additionally, add-ons may have prerequisites, which can be found in the Amazon EKS documentation or the ISV’s installation guide. In the case of Kasten K10 by Veeam, the prerequisites are:

Step 2

Declare the Terraform resource for each add-on as follows. Depending on whether the Amazon EKS third-party add-on requires custom configurations or a custom IAM role with policy, choose one of the following options to declare the add-on module in the existing cluster Terraform file.

For adding a third-party add-on without custom configurations, you can add the following resource to the existing EKS cluster module.

resource "aws_eks_addon" "example" {
  cluster_name                = "<eksclustername>"
  addon_name                  = "<addonname>"
  addon_version               = "<addonversion>" #e.g., previous version v1.9.3-eksbuild.3 and the new version is v1.10.1-eksbuild.1
  resolve_conflicts_on_update = "PRESERVE"
}

To add an add-on with custom configurations, you need to define the JSON schema obtained from Step 1 with custom configuration values.

resource "aws_eks_addon" "example" {
  cluster_name                = "<eksclustername>"
  addon_name                  = "<addonname>"
  addon_version               = "<addonversion>"
  resolve_conflicts_on_create = "OVERWRITE"

  configuration_values = jsonencode({
    replicaCount = 4
    resources = {
      limits = {
        cpu    = "100m"
        memory = "150Mi"
      }
      requests = {
        cpu    = "100m"
        memory = "150Mi"
      }
    }
  })
}

For add-ons like Kasten K10, you need an IAM role for service account with IAM role for Kasten K10 to validate license of your purchase by connecting with AWS License Manager Service. Managed policies required for each third-party add-on can be found in Additional Amazon EKS add-ons from independent software vendors. You can use Terraform to define an IAM policy document and IAM role. You can reference the ARN of the new IAM role inside the third-party add-on resource using service_account_role_arn parameter.

Diagram shows Kasten K10 Addon documentation inside Amazon EKS Addon documentation page. Diagram shows the service account name k10_k10, the eksctl command for creating the IRSA role with AWSLicenseManagerConsumption policy.

ALT TXT: Diagram shows Kasten K10 Addon documentation inside Amazon EKS Addon documentation page. Diagram shows the service account name k10_k10, the eksctl command for creating the IRSA role with AWSLicenseManagerConsumption policy.


## Kasten K10 IAM Policy
data "aws_iam_policy_document" "kasten_k10_iam_role_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(module.eks.cluster_oidc_issuer_url, "https://", "")}:sub"
      values   = ["system:serviceaccount:kasten-io:k10-k10"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(module.eks.cluster_oidc_issuer_url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }

    principals {
      identifiers = [module.eks.oidc_provider_arn]
      type        = "Federated"
    }
  }
}

## Kasten K10 IAM role
resource "aws_iam_role" "kasten_k10_iam_role" {
  assume_role_policy = data.aws_iam_policy_document.kasten_k10_iam_role_assume_role_policy.json
  name               = "kasten-k10-iam-role"
  managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy"]
}

Now to declare the Kasten K10 add-on, add a resource module in the Terraform with reference to IAM role created in last step and  EBS CSI driver add-on created earlier.

### Addon Kasten K10
resource "aws_eks_addon" "kasten_k10" {
  depends_on = [module.eks.aws-ebs-csi-driver]
  addon_name = "kasten_k10"
  cluster_name = module.eks.cluster_name
  service_account_role_arn = aws_iam_role.kasten_k10_iam_role.arn
  addon_version = "v5.5.7-eksbuild.0"
}

For EKS cluster versions greater than version 1.23, the Kasten K10 add-on has a dependency on the Amazon EBS CSI driver add-on to connect with the underlying PersistentVolumeClaims (PVCs).

Diagram shows installation details and prerequisites for Amazon EBS CSI Driver on Amazon EKS addon documentation page.

ALT TXT: Diagram shows installation details and prerequisites for Amazon EBS CSI Driver on Amazon EKS addon documentation page.

If you want take a look at the entire Terraform resource module with logic for EKS cluster creation, Amazon EBS CSI driver add-on creation, and finally, Kasten K10 add-on creation., follow this link.

On the other hand, you can either create a new IAM role for the existing cluster using the eksctl command as follows. With an IAM role for EKS cluster created outside of Terraform, you can remove or comment out the ‘## Kasten K10 IAM role’ and ‘## Kasten K10 IAM Policy’ reosurces in the Terraform module above.

eksctl create iamserviceaccount --name k10-k10 --namespace kasten-io --cluster my-cluster --role-name my-kasten-role \
    --role-only --attach-policy-arn arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy --approve

Now, you are ready to apply your changes on Terraform module.  Use the terraform init command to initialize your configurations and terraform apply to deploy your changes. Optionally, you can use terraform plan command to preview the execution plan. While applying changes using terraform apply command, You will be asked for explicit confirmation on resource creation, confirm the changes and trigger the deploy into your AWS account.  Make sure you have configured AWS credentials properly before using these Terraform commands.

terraform init

Diagram shows AWS Cloud shell cli with output terraform init command

ALT TXT: Diagram shows AWS Cloud shell cli with output terraform init command

terraform apply

Diagram shows AWS Cloudshell with results of terraform apply command with a prompt for applying or deploy changes.

ALT TXT: Diagram shows AWS Cloudshell with results of terraform apply command with a prompt for applying or deploy changes.

Terraform checks the status of the resources in AWS account for successful creation. If you watch the logs closely, you will see successfully deployment of the Kasten K10 addon in the new cluster as well.

Diagram shows AWS Cloudshell with the message from Terraform on successful creation of 62 resources.

ALT TXT: Diagram shows AWS Cloudshell with the message from Terraform on successful creation of 62 resources.

You can use below AWS cli command to describe the list of addons your newly created EKS cluster.

aws eks list-addons --cluster-name my-cluster

Optionally, You can visit the cluster in Amazon EKS console and check the addons installed under Add-ons tab.

To access the Kasten K10 dashboard, run the following command to temporarily port-forward the service to your localhost.

$ kubectl --namespace kasten-io port-forward service/gateway 8080:8000

Forwarding from 127.0.0.1:8080 -> 8000
Forwarding from [::1]:8080 -> 8000
Handling connection for 8080
Handling connection for 8080
...

You can access the Kasten K10 dashboard at http://127.0.0.1:8080/k10/#/dashboard. You may want to set up an ingress for proper access to this portal, as documented in the Kasten K10 guide.

In your browser, navigate to localhost:8080 and explore the Kasten K10 dashboard. Diagram shows the Kasten K10 dashboard with applications discovered inside the deployed cluster. The dashboard shows at a high level how many active data backup and import policies are set up. On the right-hand side, under Usage & Reports, you can see the current usage report for the amount of data that was backed up. Below that, under Activity, you can see a list of actions with the duration for each action. You can click on each one of the cards to either change the applications being managed, change policies, or look at a detailed usage report.

Kasten’s dashboard

ALT TXT:  In your browser, navigate to localhost:8080 and explore the Kasten K10 dashboard. Diagram shows the Kasten K10 dashboard with applications discovered inside the deployed cluster. The dashboard shows at a high level how many active data backup and import policies are set up.

On the right-hand side, under Usage & Reports, you can see the current usage report for the amount of data that was backed up. Below that, under Activity, you can see a list of actions with the duration for each action. You can click on each one of the cards to either change the applications being managed, change policies, or look at a detailed usage report.

To update a third-party add-on on an existing cluster using Terraform, modify the add-on resource inside the Terraform module and use the terraform apply command to deploy the changes to the EKS cluster.

Clean up

Terraform maintains the state of resources installed by it. It maintains the state of add-ons installed on an EKS cluster as well. Same can be leveraged to delete an add-on from an existing cluster.

You can issue an explicit terraform destroy command against the existing resource.

terrafrom destroy --target aws_eks_addon.kasten_k10

Conclusion

In this blog post, you learned how to add operational capabilities on your Amazon EKS cluster using third party add-ons from AWS Marketplace and infrastructure as code tool Terraform. I showed you how to create and manage third-party add-ons on your existing Amazon Elastic Kubernetes Service cluster using Terraform modules. You learned how to find the dependencies for your third party addon software before adding it to your Amazon EKS Cluster.

Next Steps

About The Author

Swaminathan Jayaraman is a Solutions Architect with AWS Marketplace. He supports buyers on procuring third party products via AWS Marketplace and sellers on listing their products successfully in AWS Marketplace. He has over 14 years of industry experience in developing and managing large scale applications, deploying SaaS solutions and supporting cloud migrations. He loves problem solving and always enjoys a good technical conversation.