AWS Security Blog

How to Detect and Automatically Revoke Unintended IAM Access with Amazon CloudWatch Events

Update on October 24, 2018: Note that if you do not author the Lambda function correctly, this setup can create an infinite loop (in this case, a rule that is fired repeatedly, which can impact your AWS resources and cause higher than expected charges to your account). The example Lambda function I provide in Step 3 ensures that you make the IAM call only if the user is not part of the admin group, eliminating the risk of a loop.

AWS Identity and Access Management (IAM) enables you to create IAM users and roles in your account, each with a specific set of permissions. For example, you can create administrative users who have access to all AWS APIs (also called actions), and you can create other users who have access to only a specific subset of those APIs. In the case of IAM APIs, only your administrative users should have access to them.

If your account is shared across departments in your organization, monitoring the permissions of your users can become a challenge as the number of users grows. For example, what if a user is granted unintended IAM API access and the user begins making API calls? In this post, I will show a solution that detects API callers who should not have IAM access and automatically revokes those permissions with the help of Amazon CloudWatch Events.

Solution overview

CloudWatch Events enables you to create rules to deliver events (including API call events) to targets such as AWS Lambda functions, Amazon Kinesis streams, and Amazon SNS topics. As the following diagram of this solution shows, (1) IAM sends API call events (including the ones that failed with a 4xx client error) to CloudWatch Events. Then, (2) a CloudWatch Events rule I create delivers all the calls to IAM APIs as events to a Lambda function. If the caller is not part of the admins group, (3) the Lambda function attaches a policy to the IAM user that denies (and effectively revokes) IAM access for the user.

Diagram of the solution

To automatically detect and revoke unintended IAM API access, I will:

  1. Create an IAM policy that will deny access to any IAM API.
  2. Create an IAM role that allows access to IAM APIs in order to use it in my Lambda function as an execution role.
  3. Create a Lambda function that will receive the API call event, check whether it represents a policy violation, and, if so, assume the role created in the previous step. The function then can make IAM calls, and attach the policy created in Step 1 to the user who should not have IAM access.
  4. Create a CloudWatch Events rule that matches AWS API call events and invokes the Lambda function created in the previous step.

Note that this process is a reactive approach (a complement to the proactive approach) for “just in case” situations. It is one way to make sure that all IAM users are created with locked-down policies and the admin group reviews any change.

The rest of this blog post details the steps of this solution’s deployment.

Important: Potential risk of an infinite loop. Note that you’re going to match IAM API events and deliver them to a Lambda function that will make other IAM API calls. If the Lambda function is not authored correctly, the setup can result in an infinite loop (in this case, a rule that is fired repeatedly, which can impact your AWS resources and cause higher than expected charges to your account). The example Lambda function I provide in Step 3 ensures that you make the IAM call only if the user is not part of the admin group, eliminating the risk of a loop. When trying this setup, make sure you monitor the number of invocations for your Lambda function.

Deploying the solution

I will follow the four solution steps in this section to use CloudWatch Events to detect and automatically revoke IAM access for a user who should not have had IAM access, starting with the creation of an IAM policy that denies access to any IAM API.

Step 1: Create an IAM policy that will deny access to any IAM API

I start by creating an IAM policy that denies access to IAM. Note that explicit Deny policies have precedence over explicit Allow policies. For further details about IAM policy evaluation logic, see IAM Policy Evaluation Logic.

First, I put the following policy document into a file named access_policy1.json.

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "DenyIAMAccess",
       "Effect": "Deny",
       "Action": "iam:*",
       "Resource": "*"
     }
   ]
}

Then, I run the following AWS CLI command. (To learn more about setting up AWS CLI, see Getting Set Up with the AWS Command Line Interface.)

$ aws iam create-policy 
--policy-name DenyIAMAccess 
--policy-document file://access_policy1.json

That is all it takes to create an IAM policy that denies access to IAM. However, if I wanted to use the IAM console instead to create the IAM policy, I would have done the following:

  1. Go to IAM console and click Policies on the left pane.
  2. Click Create Policy and click Select next to Policy Generator.
  3. On the next page:
    1. For Effect, select Deny.
    2. For AWS Service, select AWS Identity and Access Management.
    3. For Actions, select All Actions.
    4. For Amazon Resource Name (ARN), type * (an asterisk).
  4. Click Add Statement, and then click Next Step.
  5. Name the new policy, and then click Create Policy to finish.

Step 2: Create an IAM role for the Lambda function

In this step, I create an execution IAM role for my Lambda function. The role will allow Lambda to perform IAM actions on my behalf while the function is being executed.

In the Lambda function, I use the AttachUserPolicy, ListGroupsForUser, and PutUserPolicy IAM APIs, and Lambda uses CreateLogGroup, CreateLogStream, and PutLogEvents  CloudWatch Logs APIs to write logs while the Lambda function is running. I put the following access policy document in a file named access_policy2.json.

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "AllowIAMForLambdaPolicy",
       "Effect": "Allow",
       "Action": [
         "iam:AttachUserPolicy",
         "iam:ListGroupsForUser",
         "iam:PutUserPolicy",
         "logs:CreateLogGroup",
         "logs:CreateLogStream",
         "logs:PutLogEvents"
       ],
       "Resource": "*"
     }
   ]
}

Next, I put the following trust policy document in a file named trust_policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Finally, I run the following CLI commands to create my role. Here and throughout this post, remember to replace the placeholder account ID with your own account ID.

$ aws iam create-policy 
--policy-name AllowIAMAndLogsForLambdaPolicy 
--policy-document file://access_policy2.json

$ aws iam create-role 
--role-name RoleThatAllowsIAMAndLogsForLambda 
--assume-role-policy-document file://trust_policy.json

$ aws iam attach-role-policy 
--role-name RoleThatAllowsIAMandLogsforLambda 
--policy-arn arn:aws:iam::123456789012:policy/AllowIAMAndLogsForLambdaPolicy

If you would prefer to use the IAM console to create the IAM role, do the following:

  1. Create an IAM policy with the access policy document shown at the beginning of this step (the policy-creation steps are the same steps shown in Step 1).
  2. In the left pane of the IAM console, click Roles. On the next page, click Create New Role.
  3. After you name the role and click Next Step, on the Select Role Type page, click Select next to AWS Lambda. Click Next Step.
  4. On the Attach Policy page, select the new IAM policy you just created, and then click Next Step.
  5. On the Review page, review the new IAM role you created, and then click Create Role to finish.

Step 3: Create the Lambda function

In this step, I create the Lambda function that will process the event and decide whether there is a violation. An AWS API call event in CloudWatch Events looks like the following (I omitted the request and response bodies to save space).

{
   "version": "0",
   "id": "4223e154-e763-4770-8108-83b7461eb51b",
   "detail-type": "AWS API Call via CloudTrail",
   "source": "aws.iam",
   "account": "123456789012",
   "time": "2016-01-21T17:22:36Z",
   "region": "us-east-1",
   "resources": [],
   "detail": {
     "eventVersion": "1.02",
     "userIdentity": {
       "type": "IAMUser",
       "principalId": "AIDACKCEVSQ6C2EXAMPLE",
       "arn": "arn:aws:iam::123456789012:user/some-user",
       "accountId": "123456789012",
       "accessKeyId": "AKIAIOSFODNN7EXAMPLE”,
       "userName": "some-user"
     },
     ...
   }
}

I will use the following Lambda function that revokes IAM access for any user that is not part of the admins group.

var aws = require('aws-sdk');
var iam = new aws.IAM();

// ARN of the policy that denies all IAM access
var policyArn = 'arn:aws:iam::123456789012:policy/DenyIAMAccess';

// Name of the group for admin users
var adminGroupName = 'admins';

exports.handler = function(event, context) {
     // Print the incoming Amazon CloudWatch Events event.
     console.log('Received event:', JSON.stringify(event, null, 2));

     // If the caller is not an IAM user, do nothing.
     if (event.detail.userIdentity.type != 'IAMUser') {
         console.log('This call was not made by an IAM user!');
         context.done();
     } else {
         // Get the user name from the event.
         var userName = event.detail.userIdentity.userName;    

         // List the groups for the user. If the user is not part of the
         // 'admins' group, revoke their IAM access.
         iam.listGroupsForUser({UserName: userName}, function(error, data) {
             if (error) {
                 console.log(error);
                 context.fail('Could not list groups for the user!');
             } else {
                 for (var i = 0; i < data.Groups.length; i++) {
                     if (data.Groups[i].GroupName == adminGroupName) {
                         console.log('IAM user ' + userName + ' is part of the admins group.');
                         context.done()
                     }
                 }
                 revokeAccess();
             }
         });
     }

     // Revoke IAM access by attaching a Deny policy to the unauthorized
     // IAM user. If the IAM user already has more than the allowed number
     // of managed policies attached, add an inline policy to
     // deny access.
     function revokeAccess() {
         iam.attachUserPolicy(
             {UserName: userName, PolicyArn: policyArn},
             function(error, data) {
                 if (error) {
                     if (error.code == 'LimitExceeded') {
                         revokeAccessByInlinePolicy();
                     } else {
                         console.log(error);
                     }
                 } else {
                     console.log('Revoked IAM access for IAM user ' + userName);
                     context.done(); 
                 }
             }
         );
     }

     // Revoke IAM access by adding an inline policy.
     function revokeAccessByInlinePolicy() {
         var policyDocument =
             '{' +
                 '"Version": "2012-10-17",' +
                 '"Statement": [' +
                     '{' +
                         '"Effect": "Deny",' +
                         '"Action": "iam:*",' +
                         '"Resource": "*"' +
                     '}' +
                 ']' +
             '}';

         iam.putUserPolicy(
             {UserName: userName, PolicyName: 'DenyIAM', PolicyDocument: policyDocument},
             function(error, data) {
                 if (error) {
                     console.log(error);
                     context.fail('Failed to add inline DenyIAM policy!');
                 } else {
                     console.log('Revoked access via inline policy for IAM user ' + userName);
                     context.done(); 
                 }
             }
         );
     }
}

To create the preceding Lambda function, do the following:

  1. Go to the Lambda console and ensure you are in the US East (N. Virginia) Region (us-east-1). IAM is a global service and its AWS API call events are only available in that region.
  2. Click Create a Lambda Function, and then select the hello-world blueprint. (You can use the Filter box to search for the blueprint, if it does not appear on your first page of blueprints.)
  3. Give the function a name and a description. I named mine RevokeIAMAccess.
  4. Select Node.js for Runtime.
  5. Copy the code in the preceding Lambda function and paste it in the Lambda function code editor.
  6. In the Lambda function handler and role section, select from the drop-down list the IAM role you created earlier in this post in Step 2. I selected RoleThatAllowsIAMAndLogsforLambda.
  7. Increase Timeout to 30 seconds.
  8. To finish, click Next and then click Create Function.

Step 4: Create a CloudWatch Events rule

Now, I will create the CloudWatch Events rule that will be triggered when an AWS API call event is received from IAM. I will use AWS CLI to create the rule. A rule in CloudWatch Events must have an event pattern. Event patterns lets you define the pattern to look for in an incoming event. If the pattern matches the event, the rule is triggered. First, I put the following event pattern in a file named event_pattern1.json.

{
   "detail-type": [
     "AWS API Call via CloudTrail"
   ],
   "detail": {
     "eventSource": [
       "iam.amazonaws.com"
     ]
   }
}

Next, I run the following AWS CLI commands to create the CloudWatch Events rule, and add the Lambda function to it as a target.

$ aws events put-rule 
--name DetectIAMCallsAndRevokeAccess 
--event-pattern file://event_pattern1.json

$ aws events put-targets 
--rule DetectIAMCallsAndRevokeAccess 
--targets Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:RevokeIAMAccess

Finally, I run the following AWS CLI command to allow CloudWatch Events to invoke my Lambda function for me.

$ aws lambda add-permission 
--function-name RevokeIAMAccess 
--statement-id AllowCloudWatchEventsToInvoke 
--action 'lambda:InvokeFunction' 
--principal events.amazonaws.com 
--source-arn arn:aws:events:us-east-1:123456789012:rule/DetectIAMCallsAndRevokeAccess

If you would prefer to use the CloudWatch Events console to create this rule, do the following:

  1. Select the US East (N. Virginia) Region (us-east-1). Go to the CloudWatch console, and click Rules under Events in the left pane.
  2. Click Create rule. Select AWS API Call from the first Event selector drop-down list. If you see a warning that you don’t have CloudTrail events turned on, this means the AWS API call events are not available for your account in CloudWatch Events. See Creating and Updating Your Trail to enable it.
  3. Select IAM from the Service name drop-down list. Leave the Any operation option selected.
  4. Click Add target under Targets, and then select Lambda function from the first drop-down list. From the Function drop-down list, select the function you created earlier that revokes the IAM access.
  5. Leave the Matched event option selected under Configure input.
  6. Click Configure details. Name the rule and add a description of the rule. I named my rule DetectIAMCallsAndRevokeAccess.

From now on, whenever a user who was given unintended access to IAM APIs makes an API call against IAM, this setup will detect those events and automatically revoke the user’s IAM access. Note that the new IAM policy might not take effect for a few minutes after you create it, because of eventual consistency. Therefore, it is possible that a user will continue to be able to make successive calls to IAM for a short time after the first detection.

The power of the event pattern: How to reduce the number of calls to the Lambda function

Let me revisit the Lambda function I created in Step 3. It has the following If statement so that it processes only API call events originated by an API call made against IAM by a user.

// If the caller is not IAM user then do nothing.
if (event.detail.userIdentity.type!= 'IAMUser') {
    context.succeed('This call was not made by an IAM user!');
}

I can delete this statement from my Lambda function and let CloudWatch Events do the work for me by using the following event pattern in my rule.

{
   "detail-type": [
     "AWS API Call via CloudTrail"
   ],
   "detail": {
     "eventSource": [
       "iam.amazonaws.com"
     ],
     "userIdentity": {
       "type": [
         "IAMUser"
       ]
     }
   }
}

This event pattern ensures that my rule matches only AWS API call events that are made against IAM by an IAM user in my account. This will reduce the number of calls to my Lambda function. To learn more about event patterns, see Events and Event Patterns.

Conclusion

In this post, I have shown how you can help detect unintended IAM access and automatically revoke it with the help of Amazon CloudWatch Events. Keep in mind that IAM API call events and Lambda functions are only a small set of the events and targets that are available in CloudWatch Events. To learn more, see Using CloudWatch Events.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about this post or how to implement the solution described, please start a new thread on the CloudWatch forum.

– Mustafa