AWS Security Blog

How to use AWS Secrets & Configuration Provider with your Kubernetes Secrets Store CSI driver

January 2, 2024: We’ve updated this post to include the new failover Region feature.

April 29, 2021: We’ve updated the order of the commands in Step 1.

April 23, 2021: We’ve updated the commands in Steps 1 and 5 and in the “Additional Features” section.

Using AWS Secrets Manager, you can more securely retrieve secrets from Secrets Manager for use in your Amazon Elastic Kubernetes Service (Amazon EKS) Kubernetes pods. With the launch of AWS Secrets Manager and Configuration Provider (ASCP), you have a simple-to-use plugin for the industry-standard Kubernetes Secrets Store and Container Storage Interface (CSI) driver, used for providing secrets to applications that operate on Amazon EKS.

Since December 2022, Secrets Manager provides a failover AWS Region feature to use with your Kubernetes Secrets Store and CSI driver. This new feature helps you securely and seamlessly use secrets stored in another Region with your Kubernetes pods. Previously, the Secrets Store CSI driver supported mounting secrets from a single Region into a pod. This meant that if you faced connectivity issues with Secrets Manager in that Region and the secret couldn’t be mounted, you had to manually failover and mount a secret from a different Region. However, the new CSI driver failover Region removes the challenge of manually failing over to a different Region, helping to ensure a smooth transition between both Regions while still using the ability of Secrets Manager to securely store, rotate, and manage your secrets throughout their lifecycle.

With ASCP, you can securely store and manage your secrets in Secrets Manager, and retrieve them through your applications that are running on Kubernetes, without the need to write custom code. You also have the added benefit of using AWS Identity and Access Management (IAM) and resource policies to help limit and restrict access to your secrets. This tightly controls which secrets are accessible by the cluster. If you have enabled the rotation reconciler feature of the Secret Store CSI driver, ASCP will work with it to retrieve the latest secret from the secret provider. After you install and enable ASCP, it helps ensure that your applications receive the most current version of the secret when the pod starts, so that you can benefit from the lifecycle management capabilities of Secrets Manager. You not only gain the benefit of a natively-integrated secrets management solution, but also the ability to provide configurations in a single provider.

Note: The failover Region secret and API requests made to the secret are billed as a separate secret and API call.

Overview

In this post, we will walk you through how to set up ASCP to work with the Secrets Store CSI driver on your Kubernetes clusters. The Secrets Store CSI driver allows Kubernetes to mount secrets stored in Secret Manager into the pods as volumes. After the volumes are attached, the data is mounted into the container’s file system. We will also show you how to use the new failover Region feature with your CSI driver to support your disaster recovery plan.

Figure 1: Overview

Figure 1: Overview

This solution includes the following steps, which we describe in more detail in the following sections:

  1. Restrict access to your pods using IAM roles for service accounts
  2. Install the Kubernetes secrets store CSI driver
  3. Install the AWS Secrets Manager and Configuration Provider (ASCP)
  4. Create and deploy the SecretProviderClass custom resource
  5. Configure and deploy the pods to mount the volumes based on the configured secrets
  6. Load secrets and configurations from the volumes mounted to the container
  7. Configure and enable the failover Region.

Prerequisites

This solution has the following prerequisites:

Deploy the solution

Step 1: Restrict access to your pods using IAM roles for service accounts

You will use IAM roles for service accounts (IRSA) to limit secret access to your pods. By setting this up, the provider will retrieve the pod identity and exchange this identity for an IAM role. ASCP will then assume the IAM role of the pod and only retrieve secrets from Secrets Manager that the pod is authorized to access. This helps prevent the container from accessing secrets that are intended for another container that belongs to another pod.

Run the following command to turn on OpenID Connect (OIDC). Remember to replace<REGION> and <CLUSTERNAME> with your own values.

eksctl utils associate-iam-oidc-provider --region=<REGION> --cluster=<CLUSTERNAME> --approve #Only run this once

To create your service account role, run the following command to associate the policy (from the Prerequisites section) with your service account. Replace <NAMESPACE>, <CLUSTERNAME>, <IAM_policy_ARN>, <SERVICE_ACCOUNT_NAME> with your own values.

eksctl create iamserviceaccount --name <SERVICE_ACCOUNT_NAME> --namespace <NAMESPACE> --cluster <CLUSTERNAME> --attach-policy-arn <IAM_policy_ARN> --approve --override-existing-serviceaccounts

Step 2: Install the Kubernetes secrets store CSI driver

From your terminal where you have kubectl installed, run the following helm commands to install the CSI driver.

helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

Next, you need to determine whether you want to turn on automated rotation for the driver using the rotation reconciler feature, or whether you don’t need to periodically pull updated secrets.

If you don’t need to periodically pull updated secrets, initialize the driver with the following command:

helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

Note: If you are running an older version of the driver, you might need the flag –set grpcSupportedProviders=”aws”.

If you want to turn on automated rotation for the driver using the rotation reconciler feature which is currently in alpha, use the command as follows (you can adjust the rotation interval as you desire to find an appropriate balance between API call cost consideration and rotation frequency):

helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set enableSecretRotation=true --set rotationPollInterval=3600s

To validate that the installer is running as expected, run the following command:

kubectl get pods --namespace=kube-system

The output should show the following Secrets Store CSI driver pods and custom resource definitions (CRDs) deployed:

csi-secrets-store-qp9r8         3/3     Running   0          4m
csi-secrets-store-zrjt2         3/3     Running   0          4m
 
kubectl get crd
NAME                                            
secretproviderclasses.secrets-store.csi.x-k8s.io
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io

Step 3: Install the AWS Secrets Manager and Configuration Provider (ASCP)

Using the CSI driver, you can mount your secrets in your EKS Kubernetes pods. To retrieve the secrets from Secrets Manager so that the CSI driver can mount them, you need to install the AWS Secrets Manager and Configuration Provider (ASCP). You do this by running the following command in your terminal, which will pull down the installer file without the need to clone the entire repo.

curl -s https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/deployment/aws-provider-installer.yaml | kubectl apply -f -

Step 4: Create and deploy the SecretProviderClass custom resource

To use the Secrets Store CSI driver, you need to create a SecretProviderClass custom resource. This provides driver configurations and provider-specific parameters to the CSI driver itself. The SecretProviderClass resource should have at least the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws                               
  parameters:                                 # provider-specific parameters

To use ASCP, you create the SecretProviderClass to provide a few more details of how you are going to retrieve secrets from Secrets Manager. The SecretProviderClass MUST be in the same namespace as the pod that references it. The following is an example SecretProviderClass configuration:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    objects:  |
     - objectName: "MySecret2"
        objectType: "secretsmanager"

Step 5: Configure and deploy the pods to mount the volumes based on the configured secrets

Update your deployment yaml to use the secrets-store.csi.k8s.io driver, and reference the SecretProviderClass resource created previously. You should save this on your local desktop.

The following is an example of how to configure a pod to mount a volume based on the SecretProviderClass to retrieve secrets from Secrets Manager. In this example, I used NGINX. For your secret, the mount point and SecretProviderClass configuration will be in the pod deployment specification file. For another example, see ExampleDeployment.yaml. Make sure to replace <SERVICE_ACCOUNT_NAME> with your own information.

kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: aws-node ## <SERVICE_ACCOUNT_NAME>
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: mysecret2
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: mysecret2
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"

On pod start and restart, the CSI driver will call the provider binary to retrieve the secret and configurations from Secrets Manager and Parameter Store, respectively. After successfully retrieving this information, the CSI driver will mount them to the container’s file system. You can validate that the volume is mounted properly after a restart by running the following command:

kubectl exec -it nginx-secrets-store-inline -- ls /mnt/secrets-store/

You should get the following response:

MySecret2

Step 6: Load secrets and configurations from the volumes mounted to the container

Both secrets and configurations will be fetched at pod initialization during the mount operation. This can add a small amount of latency when using the native Kubernetes secrets, but it is similar to the experience of retrieving secrets through a custom or third-party tool. After initialization, your pod will not be impacted. ASCP, along with the rotation reconciler component, will update the values in the mount path and in the Kubernetes secret. The workload pods will watch the file system to track changes and automatically pick up new credentials. In the case of environmental variables, you will need to restart your pods.

Step 7: Configure and enable the failover Region

The updated CSI driver supports an automated failover feature to fetch secrets from a secondary Region. This helps with availability during connectivity outages and disaster recovery configurations. You can define the automated failover Region using the failoverRegion parameter in the SecretProviderClass.yaml file.

Additionally, you can include the failoverObject parameter. This is an optional field that specifies the secret in the failover Region. If you don’t include this, the CSI driver searches for the secret with the same secret ID in the failover Region that is specified in the objectName field.

To test the failover feature, update the SecretProviderClass.yaml file with the following content:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    region: "eu-west-2"
    failoverRegion: "eu-west-1"
    objects: |
      - objectName: "mysecret2"
        objectType: "secretsmanager"
        objectAlias: "eksblogdemo"

The preceding SecretProviderClass configuration looks for the secret named “mysecret2” in the primary Region “eu-west-2”. If that fails, it fails over to the “eu-west-1” Region and looks for the secret with the same name: “mysecret2”. The objectAlias is an optional field that specifies the name of the file under which the secret will be mounted. In the absence of the objectAlias field, the file name is the same as the value specified for objectName.

Before you apply the changes, you need to simulate the 500 internal server error using the DNS firewall rule group through the following steps:

  1. Make sure that you’re in the same Region as the EKS cluster, and navigate to the Amazon VPC console.
  2. In the left navigation pane, under DNS firewall, select Rule groups.
  3. Choose Add rule group, provide a name and (optional) description, and choose Next.
    4. Figure 2: Add rule group

    Figure 2: Add rule group

  4. On the Add rules – optional screen, choose Add rule.
    5. Figure 3: Add a rule

    Figure 3: Add a rule

  5. In the Rule details section, enter a name and optional description.
  6. In the Domain list section, select Add my own domain list and enter secretsmanager.eu-west-2.amazonaws.com in the box.
  7. In the Action section, select BLOCK as the action and NODATA as the response.
  8. Select Add rule.
    9. Figure 4: Set up rule details, domain list, and action

    Figure 4: Set up rule details, domain list, and action

  9. On the Set rule priority screen, choose Next.
  10. On the Add tags screen, choose Next.
  11. On the Review and create screen, choose Create rule group.
  12. From the console, select the rule group, and then navigate to the Associated VPCs tab.
  13. Choose Associate VPC and then choose the VPC that the EKS cluster uses.
    14. Figure 5: Associate VPC

    Figure 5: Associate VPC

  14. Choose Associate.

When the association status changes to Complete, apply the change by running the following command:

kubectl apply -f SecretProviderClass.yaml -n kube-system

Create a new deployment yaml file using the following specification and configure a pod to mount volumes based on the secretProviderClass. For another example of a deployment file, see ExampleDeployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: aws-node
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "nginx-deployment-aws-secrets"
      containers:
        - name: nginx-deployment
          image: nginx
          ports:
            - containerPort: 80
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true

Create the resources and verify that the pods were created.

kubectl apply -f nginx_deployment.yaml -n kube-system

kubectl get pods -l "app=nginx" -A

Verify that the secret has been mounted. Be sure to update the <Pod-ID> with your own information.

kubectl exec -it nginx-deployment-<Pod-ID> -n kube-system -- cat /mnt/secrets-store/eksblogdemo;echo

You should see the content of your secret, confirming that the failover was successful.

After testing the failover feature, complete the following steps to remove forced failover:

  1. In the DNS Firewall section, choose Rule groups.
  2. Select the rule group that you created previously and choose the Associated VPCs tab.
  3. Select the associated VPC and choose Disassociate.
    4. Figure 6: Select the associated VPC

    Figure 6: Select the associated VPC

  4. Enter disassociate and then choose Disassociate to confirm disassociation of the rule group from the VPC.

    5. Figure 7: Confirm disassociation of the VPC from the rule group

    Figure 7: Confirm disassociation of the VPC from the rule group

After you make these changes, the primary Region secret should be accessible.

Additional features

The CSI driver can also sync your secrets with Kubernetes secrets. To do this, use the optional secretObjects field to define the desired state of your synced Kubernetes secret objects. The volume mount is required for the sync. The following is an example SecretProviderClass custom resource that will sync a secret from Secrets Manager to a Kubernetes secret:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    region: eu-west-2
    failoverRegion: eu-west-1
    objects:  |
      - objectName: "mysecret2"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: dbusername
          - path: password
            objectAlias: dbpassword
  secretObjects:       # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
          
    - secretName: kubesecret  # name of the Kubernetes Secret object

      type: Opaque           # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls
 
      data:
        #- objectName: <objectName> or <objectAlias>
        - objectName: dbusername.   # name of the mounted content to sync. this could be the object name or the object alias
          key: db_username_01       # data field to populate
        - objectName: dbpassword    
          key: db_password_01

Supported Kubernetes secret types include the following:

  • Opaque
  • Kubernetes.io/basic-auth
  • bootstrap.kubernetes.io/token
  • Kubernetes.io/dockerconfigjson
  • Kubernetes.io/dockercfg
  • Kubernetes.io/ssh-auth
  • Kubernetes.io/service-account-token
  • Kubernetes.io/tls

You also have the option of using a pod deployment yaml to set environment variables in your deployment to reference new Kubernetes secrets. The following is an example deployment yaml that creates an environment variable from a synced Kubernetes secret:

kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: aws-node
  containers:
  - image: nginx:latest
    name: nginx
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
    env:
          - name: DB_USERNAME_01
            valueFrom:
              secretKeyRef:
                name: kubesecret
                key: db_username_01
          - name: DB_PASSWORD_01
            valueFrom:
              secretKeyRef:
                name: my-secret-01
                key: db_password_01
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aws-secrets

Conclusion

In this post, we walked you through how to set up and configure the new AWS Secrets Manager and Configuration Provider (ASCP) to work with Amazon EKS and other Kubernetes clusters that you are running and how to configure and simulate the CSI driver failover feature. By using ASCP and the new failover feature, you can provide more protection of your secrets with encryption and auto-rotation features while still meeting your disaster recover needs. This allows you to focus more on developing your applications rather than on fine-tuning their security configurations.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Secrets Manager forum or contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Author

Tracy Pierce

Tracy is a Senior Consultant, Security Specialty, for Remote Consulting Services. She enjoys the peculiar culture of Amazon, and uses that to ensure every day is exciting for her fellow engineers and customers alike. Customer Obsession is her highest priority, both internal and external. She has her AS in Computer Security and Forensics from SCTD, SSCP certification, AWS Developer Associate certification, AWS Solutions Architect Associates certificate, and AWS Security Specialist certification. Outside of work, she enjoys time with friends, her fiancé, her Great Dane, and three cats. She also reads (a lot), builds Legos, and loves glitter.

Navaneeth Krishnan Venugopal

Navaneeth Krishnan Venugopal

Navaneeth is a Cloud Support – Security Engineer II at AWS and an AWS Secrets Manager SME. He is passionate about cybersecurity and helps provide tailored, secure solutions for a broad spectrum of technical issues faced by customers. Navaneeth has a focus on security and compliance and enjoys helping customers architect secure solutions on AWS.