Simplify compliance management of multicloud or hybrid resources with AWS Config

Organizations of all sizes operate in a compliance landscape that is complex, dynamic, and evolving rapidly, facing internal requirements as well as industry or government regulations. A multicloud strategy creates additional challenges to maintain compliance policies across cloud providers.

With AWS, you can implement compliance processes faster and more easily with automation, ready-to-use templates, and built-in best practices. Compliance controls help you detect and flag suspicious and noncompliant activity so you can take action quickly.

AWS Config helps you continuously monitor configuration changes and automate compliance reporting. In this blog post, we will demonstrate how AWS Config can be used to assess, audit and evaluate configurations for your resources on AWS, on-premises and on other clouds. We’ll show an example for tracking resource configuration from Microsoft Azure through AWS Config.

Prerequisites

• • Basic understanding of deploying Azure Event Hubs, Azure Functions
  • Basic understanding of creating an AWS Config custom resource outlined in this blog post
  • Existing networking security group in Azure or be able to create a new one
  • Azure Client ID and Secret stored in AWS Secrets Manager

Solution Architecture

To record non-AWS resource configurations in AWS Config, we create an AWS config custom resource that stores the configuration record of our resource. The approach you take to consistently monitor non-AWS resources with AWS Config will depend on the capabilities of services that you want to manage for compliance. We outline two approaches you can use to monitor resources on your non-AWS environments:

1. Trigger-based approach

This approach applies to both AWS and non-AWS services that provide you a way to notify of changes made in the resource. In this approach, you centralize notifications from the service onto a central event bus, filter the events, and invoke an AWS Lambda function that records and checks the compliance of the resource.

2. Periodic evaluation approach

This approach applies to both AWS and non-AWS services that do not have a mechanism to send events that can notify of change events. In this approach, you create a schedule to invoke our Lambda function to describe a non-AWS resource using the provider resource API and then storing the results into a custom AWS Config resource. Once you enabled recording for an AWS Config resource, you can create custom rules to evaluate its compliance.

Figure 1: Solution Architecture

Solution Walkthrough

Periodic evaluation approach

Periodic evaluation requires customers to set up an Amazon EventBridge rule to collect changes made on Azure on a desired schedule to invoke an AWS Lambda function. In this example, you’ll see how you can use periodic evaluation to monitor encryption status of Azure Blog Storage blobs within a single container. This solution can be expanded for other use cases such as monitoring encryption status of blobs within all containers or any use case where you have access to an API.

Step 1: Create an AWS Config custom resource for your Azure Blob

Please follow this blog post to configure AWS Config custom resource for your Blob Container in Azure.

In this example, you should have packaged up the Azure SDK azure.storage.blob and added it as a layer to a Lambda function. Using the Azure SDK, you will describe the schema for an Azure Blob. Then using the list_blobs() method and iterating through that list to describe the attributes for a single blob.

Result of describing the Azure Blob storage:

{
 'name': 'samplemysqltables.sql',
 'container': 'demo-container',
 'snapshot': None,
 'version_id': None,
 'is_current_version': None,
 'blob_type': <BlobType.BLOCKBLOB: 'BlockBlob'>,
 'metadata': {}, 'encrypted_metadata': None,
 'last_modified': datetime.datetime(2023, 5, 27, 2, 40, 24, tzinfo=datetime.timezone.utc),
 'etag': 'ETAG-VALUE',
 'size': 10391,
 'content_range': None,
 'append_blob_committed_block_count': None,
 'is_append_blob_sealed': None,
 'page_blob_sequence_number': None,
 'server_encrypted': True,
 'copy': {'id': None, 'source': None, 'status': None, 'progress': None, 'completion_time': None, 'status_description': None, 'incremental_copy': None, 'destination_snapshot': None},
 'content_settings': {'content_type': 'application/octet-stream', 'content_encoding': None, 'content_language': None, 'content_md5': bytearray(b'\x98!\xef\x1e\xb9\xd7\x991\x12\xfb\x8d\xc1\x84{\x83A'), 'content_disposition': None, 'cache_control': None},
 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None},
 'blob_tier': 'Hot',
 'rehydrate_priority': None,
 'blob_tier_change_time': None,
 'blob_tier_inferred': True,
 'deleted': None,
 'deleted_time': None,
 'remaining_retention_days': None,
 'creation_time': datetime.datetime(2023, 5, 27, 2, 40, 24, tzinfo=datetime.timezone.utc),
 'archive_status': None,
 'encryption_key_sha256': None,
 'encryption_scope': None,
 'request_server_encrypted': None,
 'object_replication_source_properties': [],
 'object_replication_destination_policy': None,
 'last_accessed_on': None,
 'tag_count': None,
 'tags': None,
 'immutability_policy': {'expiry_time': None, 'policy_mode': None},
 'has_legal_hold': None,
 'has_versions_only': None
}

Now that you understand what type of attributes you can collect for an Azure Blob storage you can either choose to record all the attributes or filter and just choose the attributes that are relevant to you use case. To keep the example simple let’s just pick out the attributes regarding the blob name and server encryption status. The attribute server_encrypted tells us if the object has server encryption and name tell us the blob name. Now you can create this schema and register a custom AWS Config resource for an Azure Blob with the attributes you chose in AWS CloudFormation registry. Refer to the blog post (AWS Config custom resource) to register a custom AWS Config resource.

Step 2: Create an AWS Lambda function to collect resource configuration

The Lambda function will use the Azure SDK to describe an Azure Blob storage and then use the AWS Config API to store the resource details into our custom AWS Config resource.

You can use this sample Python Code for your AWS Lambda function and this sample IAM Policy for the AWS Lambda function execution role policy.

Step 3: Create an Amazon EventBridge rule to run the AWS Lambda function on a desired schedule

Figure 2: Periodic trigger through Amazon EventBridge

After the Lambda function gets invoked, you should be able to see the configuration item under your custom resource in the AWS Config console.

Figure 3: AWS Config configuration item record

Step 4: Create a CloudFormation Guard rule to evaluate blob compliance (Optional)

For compliance purposes as an example, you can create a rule to check if we have server side encryption enabled by evaluating if server_encrypted parameter is set to true using CloudFormation Guard in the AWS Config console.

Figure 4: Compliance rules through CloudFormation Guard.

In summary, you set up a custom AWS Config resource and defined its schema, triggered a Lambda function on a schedule to update the resource’s configuration details, and defined an AWS Config CFN guard rule to evaluate the blob configuration for compliance.

Figure 5: Recorded configuration item with a compliance check.

Trigger based approach

In this example, you use AWS Config to maintain and monitor the compliance status of an Azure Network Security group. To do so, use Azure Event Hubs to centralize events across required network security group. Using Event Hub, you trigger an Azure function to filter events from our network security group to only look for events that are of type Microsoft.Network/networkSecurityGroups/write. If the event is of this type, you then invoke a Lambda function with the resourceId of the Azure network security group into the record configuration item of the Config resource.

Step 1: Create an AWS Config custom resource for your Azure Network Security group

Please follow this blog post or the AWS Config documentation to configure the AWS Config custom resource for your Network Security group in Azure. You can describe the schema for your resource with the below Azure cli or Azure Cloud Shell with the command below:

az network nsg show --name <YOUR AZURE SECURITY GROUP NAME> --resource-group <YOUR AZURE RESOURCE GROUP FOR THE SECURITY GROUP>

The output for the above command should be a json similar to this:

{  
	"etag": "W/\"etag-value\"",
	"id": "/subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Network/networkSecurityGroups/network-security-group-name",
		"location": "westus",
		"name": "network-security-group-name",
		"resourceGroup": "resource-group-name",
		"securityRules": [
			{
				"access": "Allow",
				"description": "Rule description",
				"destinationAddressPrefix": "*",
				"destinationPortRange": "22",
				"direction": "Inbound",
				"name": "SSH",
				"priority": 100,
				"protocol": "Tcp",
				"sourceAddressPrefix": "192.168.0.0/24",
				"sourcePortRange": "*"
			},
			{
				"access": "Deny",
				"description": "Rule description",
				"destinationAddressPrefix": "10.0.0.0/16",
				"destinationPortRange": "*",
				"direction": "Outbound",
				"name": "BlockOutbound",
				"priority": 200,
				"protocol": "*",
				"sourceAddressPrefix": "*",
				"sourcePortRange": "*"
			}
		],
		"subnets": [
		{
			"id": "/subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/virtual-network-name/subnets/subnet-name"
		}
	],
	"tags": {},
	"type": "Microsoft.Network/networkSecurityGroups"
}

Step 2: Centralize change notification events

In a real-world scenario, you would be collecting events from multiple resources. It is therefore, important that you aggregate these notifications into a centralized event bus which can be utilized for downstream processing. You can choose to centralize all of your notification events, only certain resources, certain resource type (e.g. Network Security Group) etc. To create an event bus in Azure, leverage Azure Event Hubs. For step 2, please Create an Azure Event Hub in your Azure Portal.

Step 3: Direct change notifications to centralized bus

Azure activity logs will log addition/deletion of a rule in your network security group. A network security group rule change logs an Administrative category activity. You now export the Network security group logs to the Azure Event Hub you created in Step 2:

• Navigate to your network security group dashboard. On the left hand panel, choose Activity Log. From the left hand panel, choose Export Activity Logs.
• Under Subscription, choose your subscription. Under Diagnostic settings, select +Add diagnostic setting
• On the Diagnostic setting page, under Logs, for Categories check Under Destination details, choose Stream to an event hub and provide your details of Event Hub you created in Step 2.

You can turn on activity logs for multiple network security groups and send them to one single Event Hub thereby consolidating your event notification mechanism.

Step 4: Create a Lambda function to record configuration item into AWS Config:

1. Create a Lambda function that will be triggered by the Azure Function we’ll creat in the next step to record configuration resource details of your Azure Network Security Group into your AWS Config custom resource.
   You can use this sample Python code for this Lambda function and this sample IAM policy for your Lambda function execution role permission.

Step 5: Invoke your Lambda function for selected change events

You’ll use an Azure Function to filter and monitor change events for only the resources you’re interested in monitoring the compliance. For an Azure network security group rule, the event you are looking for is networkSecurityGroups. For this example this Azure Function will also extract the subscriptionId, resourceGroupName and resourceName of the security group, but you can customize what data you’ll send to AWS Config based on your requirements:

• Create an Azure function with Azure Event Hub trigger.
• You can use this Python code snippet in your Azure function to extract subscriptionId, resourceGroupName and resourceName from Activity logs before sending to AWS. Remember to also add the following Python packages (azure-functions, requests, requests-aws4auth) to your txtwhen publishing your function. You can follow the documentation here.

This Azure function will invoke your Lambda function you created in the step 4.

You can invoke your Lambda function directly through a function URL HTTP(S) endpoint or using AWS SDK. In our example code above, you’re using a function URL endpoint from Lambda with AWS_IAM authentication.

Step 6 Create a Lambda function to evaluate compliance of the configuration item (Optional):

1. Optionally you can also create another Lambda function that will evaluate the compliance of the configuration item resource against a specific compliance rule your organization might have and trigger it from a Config Custom Rule.
   You can use this sample Python code for your Lambda function and this sample IAM policy for the Lambda execution role permission.
   This Lambda function will check if the Azure Security Group has any security rule with port 22 open to the internet and reports as noncompliant.

Create your AWS Config Custom Rule and use the ARN of the Lambda function you created above as a target.

This setup will now record your Azure Network Security Group configuration and check its compliance any time there’s a configuration change in your resource:

Figure 6: Configuration data for Azure Network Security Group in AWS Config

Figure 7: Compliance result for Azure Network Security Group in AWS Config

You can also check the full configuration item data as a JSON in the console or retrieve it through the AWS API:

Figure 8: Recorded configuration item (CI) for Azure Network Security Group

Conclusion

AWS Config continually assesses, audits, and evaluates the configurations and relationships of your resources on AWS, on premises, and for hybrid or multicloud resources.

In this post, we demonstrated how you can use AWS Config to assess, audit and, evaluate configurations for your resources in a multicloud context.

To learn more about AWS Config please visit the AWS Config documentation and check out our AWS Cloud Operations Competency Partners that can help customers set up, build, migrate, and operate securely and efficiently across AWS, hybrid and multicloud use cases.

About the Authors: