Integrating LDAP/AD Users to Kubernetes RBAC with the AWS-IAM-Authenticator Community Project

Aug. 30, 2022 — This blog post was written prior to AWS renaming AWS Single Sign-On. The new name is AWS IAM Identity Center (successor to AWS Single Sign-On). Please consider all references, in this blog, to AWS Single Sign-On or AWS SSO as being AWS IAM Identity Center.

Our customers ask us how they can secure access to their Amazon Elastic Container Service for Kubernetes (Amazon EKS) or Kubernetes on EC2 clusters. Our enterprise customers have implemented Active Directory (AD), Active Directory Federated Services (ADFS), or Lightweight Directory Access Protocol (LDAP) for identity and access management on-premises, and use AWS Identity and Access Management (IAM) in the AWS cloud. Many of these customers have adopted Kubernetes on-premise and want to use a common role-based access control (RBAC) policy for controlling access to Kubernetes resources, both on-premise and in AWS Cloud.

This prompted our investment in an open source initiative (initially driven by Heptio) called aws-iam-authenticator, to develop a tool that uses AWS IAM credentials to authenticate to a Kubernetes cluster. The Amazon EKS open source team uses the AWS IAM Authenticator as a starting point to map AWS IAM to Kubernetes RBAC. AWS and Heptio engineers are core maintainers of the project, and worked with a set of exceptional community engineers to donate the project to the Cloud Provider Special Interest Group.

Whether you use Amazon EKS, Kubernetes on EC2, or Kubernetes on-premises, you can declare the actions authorized for AWS IAM users in the AWS cloud or AD users on-premise using Kubernetes RBAC objects and role bindings. After the declaration, you can authenticate the AWS managed IAM user roles or AD user roles to a common Kubernetes RBAC role, and enable end-to-end authentication and authorization on your Kubernetes clusters.

In this post we will show you how to use AWS Single Sign-On (SSO), AWS Managed Microsoft Active Directory Service, and the AWS IAM authenticator to control access to your Amazon EKS cluster running in the AWS cloud. This also works with a self-managed Kubernetes cluster created with kops on AWS. We chose to leverage AWS managed services to simulate third-party identity and access management tools that you may be using on-premises, and then federate access to AWS Identity and Access Management (IAM).

To implement this approach, you will use the following services:

• AWS Organizations: Enables a single master AWS account to manage IAM roles and policies inside of managed accounts.
• AWS Single Sign-On: Integrates with Microsoft Active Directory through AWS Directory Service, allowing users to access the AWS Command Line Interface with a set of temporary AWS credentials.
• AWS Managed Microsoft AD: Provides a fully-managed Microsoft Active Directory to allow management of your users and credentials. This component can be swapped out with your own AD implementation leveraging connectors and/or federation.
• AWS Identity and Access Management (IAM): Provides native identity and access management on AWS cloud for AWS customers/users.
• AWS IAM Authenticator for Kubernetes: Implements a tool to map AWS IAM user credentials to Kubernetes identities so users familiar with AWS IAM do not need to maintain a separate set of Kubernetes credentials.

For detailed descriptions of other unmanaged solutions for federated access with identity providers that support SAML 2.0, please refer to Enabling Federation to AWS Using Windows Active Directory, ADFS, and SAML 2.0 and How to Implement a General Solution for Federated API/CLI Access Using SAML 2.0.

Prerequisites

Before you get started, be sure that:

1. You have two AWS Accounts, Account A (master) and Account B (member).
2. You have set up the AWS Organizations Service in Account A (master) as per the AWS SSO prerequisite requirements.
3. You have added Account B (member) to the organization created in Account A (master).
4. You have set up an AWS Managed Microsoft AD in Account A (master) link in us-east-1.
5. You have a deployed a Windows EC2 instance to manage the AWS Microsoft AD, and joined it to the AD domain. Ensure that you have the AD DS and AD LDS Tools installed.
6. You have enabled AWS SSO from the AWS management console.
7. You have connected AWS SSO to the Microsoft AD.
8. You have an Amazon EKS cluster deployed in Account B (member) in any available region.
9. Optional: You have mapped attributes in AWS SSO to attributes in your connected directory.

Note: If you are using an on-premises AD, you will need to perform an additional step before you can establish SSO access to AWS accounts in step 7 above. For that step, you can either:

• Create a two-way trust relationship between the AWS Managed Microsoft AD (created in step 4 above) and an on-premise AD.

OR

• Create an AD Connector in AWS to redirect requests to your on-premise AD.

Bringing It All Together

In the next few sections, we will map the AD groups to AWS IAM roles via AWS SSO AssumedRoles. We will then use Kubernetes configMaps and roles to map the associated IAM roles to Kubernetes RBAC roles. This will be done by leveraging the open source AWS IAM Authenticator to pass an IAM identity from kubectl.

For illustrative purposes, we will provide users in the AWS-EKS-Admins group full access to the EKS cluster (similar to the permissions assigned to the default cluster-admin role within Kubernetes) and we will provide users in the AWS-EKS-Dev group readOnly access to certain Kubernetes resources. The mapping is summarized in the table below:

For information on how RBAC within Kubernetes works, please read the Kubernetes reference documentation.

1. Create Users/Groups in Microsoft AD

We will create two AD groups (AWS-EKS-Prod and AWS-EKS-Dev) in the Microsoft AD. We will also create a devuser and an adminuser, and add the users to the AWS-EKS-Admins and AWS-EKS-Dev groups respectively.

Perform the following from the Windows EC2 instance that has joined the AD Domain:

• Navigate to the Active Directory Users and Computers mmc console. Alternately, you can use powershell shown as reference below.
• Create an AWS-EKS-Admins group in the AD domain.

PS C:\Users\Admin> New-ADGroup -Name "AWS-EKS-Admins" -SamAccountName AWS-EKS-Admins -GroupCategory Security -GroupScope Global -DisplayName "AWS-EKS-Admins" -Path "OU=test,DC=test,DC=eks,DC=Com" -Description "Members of this group are AWS EKS Administrators"

• Create an AWS-EKS-Dev group in the AD domain.

PS C:\Users\Admin> New-ADGroup -Name "AWS-EKS-Dev" -SamAccountName AWS-EKS-Dev -GroupCategory Security -GroupScope Global -DisplayName "AWS-EKS-Dev" -Path "OU=test,DC=test,DC=eks,DC=Com" -Description "Members of this group are AWS EKS Developers"

• Create an adminuser

PS C:\Users\Admin> $password = "EKSRulz!" | ConvertTo-SecureString -AsPlainText -Force
PS C:\Users\Admin> New-ADUser -Name adminuser -GivenName adminuser -Surname adminuser -UserPrincipalName adminuser@test.eks.com -Path "OU=Users,OU=test,DC=test,DC=eks,DC=Com" -AccountPassword $Password -ChangePasswordAtLogon $True -Enabled $True

• Create a devuser

PS C:\Users\Admin> $password = "EKSRulz!" | ConvertTo-SecureString -AsPlainText -Force
PS C:\Users\Admin> New-ADUser -Name devuser -GivenName devuser -Surname devuser -UserPrincipalName devuser@test.eks.com -Path "OU=Users,OU=test,DC=test,DC=eks,DC=Com" -AccountPassword $Password -ChangePasswordAtLogon $True -Enabled $True

• Add adminuser to the AWS-EKS-Admins group

PS C:\Users\Admin> Add-ADGroupMember -Identity AWS-EKS-Admins -Members adminuser

• Add devuser to the AWS-EKS-Dev group

PS C:\Users\Admin> Add-ADGroupMember -Identity AWS-EKS-Dev -Members devuser

2. Assign the AD Groups to the AWS SSO

Now that we have created the AD Users/Groups, we will assign the groups (and hence users) access to AWS Account B (where the EKS cluster resides). We will also add a permission set that allows Account A (master) to assume a role in Account B (member) and thereby access resources in that account.

Note that no specific permissions are assigned to the assumed role for the AD Users/Groups at this stage. By default the assumed role will not have permission to perform any operations. The specific authorization permissions will be defined via Kubernetes RBAC in the next section.

Navigate to AWS SSO Console in Account A (master) and perform the following:

• Select the AWS Accounts tab on the left panel and select AWS account A to assign users. Click Assign Users.
• Search for the AD groups AWS-EKS-Dev and AWS-EKS-Admins. Click Next: Permission sets.

• Select Create new permission set and, on the next page, select the Create a custom permission set. Give the permission set a name e.g. “AD-EKS-Dev-AssumedRole” and select the Create a custom permissions policy checkbox. Copy the following permissions policy into the window and then click Create. The following policy allows the IAM user in Account A (master) to assume any role in any account that trusts the user’s account. In response to the AssumeRole API call, a set of temporary security credentials to access AWS resources is returned by AWS STS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "*"
        }
    ]
 }

• Select the AD-EKS-Dev-AssumeRole checkbox and click Finish
• Repeat the process for the AWS-EKS-Admins group by assigning a similar permission set to the group.

Behind the scenes, AWS SSO performs the following operations in Account B (member):

• Sets up SAML federation by configuring an Identity Provider (IdP) in AWS IAM. The Identity Provider enables the AWS account to trust AWS SSO for allowing SSO access.
• Creates an AWS IAM role and attaches the above permission set as a policy to the role. This is the role that AWS SSO assumes on behalf of the Microsoft AD user/group to access AWS resources.

3. Configure Role Permissions via Kubernetes RBAC

At this point, we have set up Microsoft AD to control authentication to the AWS SSO user portal. We have also set up partial authorization by specifying which AD group has been assigned access to Account B (where the EKS cluster resides). However, access control (i.e., the level of permissions granted to the AD users) has not yet been specified. In this section, we will demonstrate how Kubernetes RBAC can be configured to define access control for AD users/groups via federation through an AWS IAM role.

Before proceeding with the steps below, be sure that that:

1. You have a working Kubernetes cluster with worker nodes.
2. You have already configured kubeconfig to manage the cluster via the cluster admin (creator).

Note: If you are using Amazon EKS, please see Getting Started with Amazon EKS for detailed steps to configure your Amazon EKS cluster.

Create a Demo Namespace

First, let’s create a demo namespace called demo-service. We will use this namespace to illustrate full and read-only access control via Kubernetes RBAC for the AWS-EKS-Admins and AWS-EKS-Dev AD groups.

$ kubectl create namespace demo-service

Next, let’s create the associated Kubernetes Roles, one each, for the AWS-EKS-Dev and AWS-EKS-Admins groups.

Create ClusterRoles

$ cat role.yaml

# role.yaml
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: demo-service:ad-cluster-admins
   namespace: demo-service
 rules:
 - apiGroups: ["*"]
   resources: ["*"]
   verbs: ["*"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: demo-service:ad-cluster-devs
   namespace: demo-service
 rules:
 - apiGroups: [""]
   resources: ["services", "endpoints", "pods", "deployments", "ingress"]
   verbs: ["get", "list", "watch"]

Next, we will apply these Roles to our cluster to enable them.

$ kubectl apply -f role.yaml

Modify Authenticator Config Map

Note: IAM Roles created by AWS IAM Identity Center will have a different suffix in their ARN if they are deleted and recreated. If your aws-auth config map and role binding only references roles created by IAM Identity Center, you may lose access to your cluster if those roles are deleted and recreated. To avoid losing access your Amazon EKS cluster, we also recommend adding an IAM role not managed by IAM Identity Center to your aws-auth config map and role binding. This IAM role can be configured to only be assumable by your IAM Identity Center roles. For more information about IAM roles please see this documentation and this blog post.

Now, let’s modify our authenticator config to allow EKS access to these groups. You will need to get the role arns for the roles that IAM assumes for the two respective AD groups, AWS-EKS-Dev and AWS-EKS-Admins. You can do that using the following command:

$ aws iam list-roles | grep Arn | grep AD-EKS-

Next, you will need to edit the aws-auth ConfigMap:

$ kubectl edit configmap aws-auth --namespace kube-system

This command will open the file in your editor. We can then add the following to the mapRoles section. Make sure to:

1. For the rolearn be sure to remove the /aws-reserved/sso.amazonaws.com/ from the rolearn url, otherwise the arn will not be able to authorize as a valid user.
2. Make sure that the groups match the —group parameter you specify in the role bindings below.

# ... mapRoles config
 - rolearn: arn:aws:iam::141548511100:role/AWSReservedSSO_AWS-EKS-Admins_b7e6a177cfc720e6
   username: adminuser:{{SessionName}}
   groups:
   - demo-service:ad-cluster-admins
 - rolearn: arn:aws:iam::141548511100:role/AWSReservedSSO_AWS-EKS-Dev-AssumedRole_7f87046f4b89cf97
   username: devuser:{{SessionName}}
   groups:
 - demo-service:ad-cluster-devs

Once you have added that, save and close this file.

Create Role Bindings

Finally, we can bind these roles to the relative groups in Kubernetes who should have access to the demo-service namespace. The AWS-EKS-Admin role will have full access and the AWS-EKS-Dev role will have read-only access to the demo-service namespace. In the commands below, we create rolebindings operations-team-binding, and development-team-binding, which assigns the Role demo-service:ad-cluster-admin and the Role demo-service:ad-cluster-devs to the Kubernetes groups demo-service:ad-cluster-admins and demo-service:ad-cluster-devs in the demo-service namespace.

Role binding for demo-service:ad-cluster-admins:

$ kubectl create rolebinding operations-team-binding --role demo-service:ad-cluster-admins --group demo-service:ad-cluster-admins --namespace demo-service

Role binding for demo-service:ad-cluster-devs:

$ kubectl create rolebinding development-team-binding --role demo-service:ad-cluster-devs --group demo-service:ad-cluster-devs --namespace demo-service

Modifying kubeconfigs

Now we can create new KUBECONFIG files for each set of role bindings, one for dev users and one for admin users. Copy your kubeconfig file and then modify the user section for your eks cluster to look like this.

~/.kube/config-test-eks-cluster-1-dev

- name: test-eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
      - "token"
      - "-i"
      - "test-eks-cluster"
      - "-r"
      - "<RoleArn of AWS-EKS-Dev AssumedRole>"

~/.kube/config-test-eks-cluster-1-admins

- name: test-eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
      - "token"
      - "-i"
      - "test-eks-cluster"
      - "-r"
      - "<RoleArn of AWS-EKS-Admins AssumedRole>"

Note: For the -r parameter above, specify the full rolearn in this case, including the path in the rolearn.

Example:

arn:aws:iam::1234567890:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AWS-EKS-Admins

Testing That All This Works

Let’s try issuing requests to the apiserver to validate that end-to-end authentication and authorization are working. To authenticate as an AD user, obtain the environment variables for that user from the AWS SSO portal. Access the portal url as follows:

Next:

1. Sign into the AWS SSO User portal with a valid AD user, e.g. adminuser, that is a member of Microsoft AD AWS-EKS-Admins group.
2. Click the Command line or programmatic access link to retrieve the environment variables for that user.
3. Export the environment variables to the terminal.

Also, make sure that the associate kubeconfig contexts are set when running the tests below:

Example as adminuser:

$ kubectl run snowflake --image=kubernetes/serve_hostname --replicas=2 --namespace=demo-service
deployment.apps "snowflake" created

$  kubectl get pods --namespace=demo-service

NAME                         READY      STATUS    RESTARTS  AGE
snowflake-54fccfcd67-9k8dq   0/1        Pending   0         3m
snowflake-54fccfcd67-qq59k   0/1        Pending   0         3m

$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "adminuser-1529910544824263459" cannot list pods in the namespace "default"

Example as devuser:

$ kubectl run snowflake --image=kubernetes/serve_hostname --replicas=2 --namespace=demo-service
Error from server (Forbidden): deployments.extensions is forbidden: User "devuser-1529910185702205240" cannot create deployments.extensions in the namespace "demo-service"

$ kubectl get pods --namespace=demo-service
NAME                         READY     STATUS       RESTARTS  AGE
snowflake-54fccfcd67-9k8dq   0/1       Pending      0         1m
snowflake-54fccfcd67-qq59k   0/1       Pending      0         1m

$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "devuser-1529910584505925203" cannot list pods in the namespace "default"

As expected, an adminuser has full permissions in the demo-service namespace, but no access in other namespaces. Whereas, a devuser has read-only access in the demo-service namespace.

Conclusion

In this post we showed how an identity in AWS Microsoft Active Directory can assume an AWS IAM role via AWS SSO to authenticate using the AWS CLI. Subsequently, the AWS IAM role can be mapped to Kubernetes RBAC via K8s configMap, clusterRole and rolebinding to authorize the active directory user with access permission in kubernetes namespaces.

You can learn more about the project here: AWS-IAM-Authenticator and we welcome your feedback and contributions.