Audit and visualize ephemeral EC2 instances using AWS CloudTrail Lake as a zero-ETL data source in Amazon Athena

Today, we are happy to announce that AWS CloudTrail Lake data is now available for zero-ETL analysis in Amazon Athena. AWS CloudTrail Lake is a managed data lake for capturing, storing, accessing, and analyzing user and API activity on AWS for audit, security, and compliance purposes. CloudTrail Lake allows you to easily aggregate activity logs across your AWS accounts through integration with AWS Organizations.  Amazon Athena allows you to query data stored in Amazon Simple Storage Service (S3) and other data sources making it a powerful tool for compliance and security related activities. Now that CloudTrail Lake tables are shareable via AWS Glue Data Catalog you can use Amazon Athena to query CloudTrail Lake logs along with other data sources as part of your compliance reporting or security investigations. Using CloudTrail Lake for zero-ETL analysis from Athena removes the operational complexities of building data processing pipelines to be able to correlate activity logs with application logs or Cost and Usage data stored in S3.  Replicating or moving data through the pipeline will no longer be necessary when using zero-ETL analysis, which will save on data processing costs as security engineers investigate security incidents or compliance officers audit resource compliance.

Zero-ETL analysis of CloudTrail Lake logs allows you to correlate and visualize compliance, cost, and usage data using Amazon QuickSight and Amazon Managed Grafana.  AWS Lake Formation gives you fine grained access control to restrict or grant specific users and roles cross account query access to be able to query the CloudTrail Lake tables shared with Glue Data Catalog.  With CloudTrail Lake also announcing a new one-year extendable retention pricing option, you are able to audit and report on our security and compliance posture at a reduced cost and level of complexity.  Other recent additions to CloudTrail Lake capability include importing existing CloudTrail logs into an existing or new event data store or you can also visualize top CloudTrail event trends with CloudTrail Lake dashboards. In addition to CloudTrail events, CloudTrail Lake also supports ingestion and analysis for additional auditable data sources such as AWS Config configuration items and 3rd party auditable activity events.

In this blog post we will show you how you use CloudTrail Lake to report and visualize compliance, activity, and cost for ephemeral Amazon Elastic Compute Cloud (EC2) instances at cloud scale.  An ephemeral workload is a temporary use of computing resources that are loaded and run when needed. Examples include Amazon EC2 Spot Instances, Amazon EMR jobs, and AWS Auto Scaling.  Each time one of our ephemeral instances is instantiated, the instance will generate CloudTrail activity which will give you information to identify the instance that was launched and user identity of who it was launched by.  As these ephemeral instances spin up and spin back down, CloudTrail Lake is capturing activity you need for auditing the activity in the future. By leveraging the autoscaling-launch-template managed rule in AWS Config you can track the definition of EC2 instances launched using an AutoScaling Group to verify their configuration at launch.  When you create a launch template, you can define a template that meets your compliance standards and then ensure that all instances are launched from this template (or other compliant templates) via AWS Config.  The autoscaling-launch-template rule evaluates the AWS::AutoScaling::AutoScalingGroup resource which captures data such as the launch template ID and instance ID information about instances launched by the auto scaling group. Using this method, you can ensure that your ephemeral instances, (1) are launched from a compliant template that you can reference for compliance reporting, (2) are having activity tracked in CloudTrail Lake, (3) are able to be reported on throughout their lifecycle from definition to instantiation and termination.  Lastly, you can correlate cost data to understand how much total spend is being impacted by noncompliant instances

Overview of solution

In this solution, you will be using AWS CloudTrail Lake, Amazon S3, Amazon EC2, AWS Cost & Usage report (CUR), and Amazon Athena for compliance reporting. You will investigate if all the instances in your environment are launched using a specific EC2 launch template and what is the financial impact of launching instances directly through AWS Console. To do so, you will create an event data store in CloudTrail Lake that aggregates and records API activity. Then you will walk through how to add an event data store as a source for Amazon Athena using Lake query federation. Finally, you correlate the CUR data with the CloudTrail activity for your EC2 instances with Amazon Athena.

To measure the cost involved with EC2 instances not launched through Launch templates, you track the RunInstances API call recorded by CloudTrail. The CUR data identifies EC2 instance ID under the field line_item_resource_id. By leveraging the CloudTrail Lake query federation, you will correlate data in CloudTrail Lake with the CUR data. This allows you to put together an audit trail for our instances that will track how they were defined, when they were launched, and how much each instance costs to run.

Prerequisites

For this walkthrough, you should have the following prerequisites:

1. An AWS account
2. Follow the steps to add a managed rule and add the rule autoscaling-launch-template
3. You should have an AWS Cost and Usage Report publishing your billing reports to an Amazon S3 bucket. If you do not, please follow the steps mentioned in this documentation to create a cost and usage report with Amazon Athena integration
4. For visualization, you should have Amazon QuickSight setup. If you do not, please follow the steps mentioned in this documentation to setup Amazon QuickSight

Create an event data store with Lake query federation

In this section, you will Create an event data store with CloudTrail Lake query federation option enabled to collect and store management events. This event data store will record RunInstances API call.

1. 1. Navigate to the CloudTrail console. From the navigation pane, under Lake, choose Event data stores.
   2. From the right panel, Choose Create event data store.
   3. On the Configure event data store page, in General details, enter a name such as “demo-cloudtrail-lake”.
   4. Specify the Pricing option as per your requirement and choose Retention Period.
   5. Under Lake query federation, choose Enabled. Under Choose IAM role, select Create and use a new role.

6. Keep the rest as default. Select Next.
7. On the Choose Events page, keep defaults. Select Next. On the Review and Create page, choose Create event data store.

Using CloudTrail Lake query federation, the CloudTrail Lake metadata table is now automatically configured in AWS Glue data catalog

To manage access permission queries from Athena, the feature has also added the event data store as a location in Lake Formation.

Using Lake Formation integration with CloudTrail Lake, you can now securely share the data in your event data store across multiple AWS accounts. More information about cross-account data sharing in Lake Formation is available in this public documentation.

Use Amazon Athena to correlate AWS CloudTrail event data with your Cost and Usage report

In this section, you will join data in CloudTrail Lake with the Cost and Usage Reports that are being delivered to S3 bucket. Please note it can take up to 24 hours for AWS to start delivering AWS Cost and Usage Reports to your Amazon S3 bucket. After delivery starts, AWS updates the AWS Cost and Usage Reports files at least once a day. This is why you might see a delay in the cost data.

CloudTrail captures the launch template ID in the requestParameters section of the RunInstance API call. The instance ID is captured in the responseElements section of the same API call. In the CUR data, the instance ID is provided in the line_item_resource_id field and the unblended cost tracked as line_item_unblended_cost. To correlate data, please follow the steps below:

1. 1. Navigate to Amazon Athena Console.
   2. From left hand panel, select Query editor. Note the database aws:cloudtrail and table created by CloudTrail Lake query federation under Tables. Here, please note that the table name is the event data store ID in AWS CloudTrail Lake.

1. 3. Under Editor, run the below query. Be sure to replace “enter-your-cur-database”.”enter-your-cur-table”, “aws:cloudtrail”.”enter-eds-id-here”, and ‘enter-approved-launch-template-id-here’ with the appropriate values:

      SELECT CostAndUsageReport.Instance_ID, CloudTrail.User_ARN, CostAndUsageReport.Unblended_COST, CostAndUsageReport.Description_COST
      FROM (
      		SELECT line_item_resource_id as Instance_ID,
      			line_item_product_code,
      			line_item_line_item_description as Description_COST,
      			SUM(line_item_unblended_cost) AS Unblended_COST
      		FROM "enter-your-cur-database"."enter-your-cur-table"
      		WHERE line_item_product_code = 'AmazonEC2'
      			AND line_item_line_item_type NOT IN ('Tax', 'Refund', 'Credit')
      		GROUP BY line_item_resource_id,
      			line_item_product_code,
      			line_item_line_item_description
      		ORDER BY Unblended_COST DESC
      	) AS CostAndUsageReport
      	JOIN (
      		SELECT useridentity.arn as User_ARN, json_extract_scalar(
      				element_at(
      					CAST(responseelements AS map(varchar, varchar)),
      					'instancesSet'
      				),
      				'$.items[0].instanceId'
      			) as instance_id
      		FROM "aws:cloudtrail"."enter-eds-id-here"
      		WHERE (
      				useragent like 'autoscaling.amazonaws.com'
      				and NOT(
      					json_extract_scalar(
      						element_at(requestparameters, 'launchTemplate'),
      						'$.launchTemplateId'
      					) like 'enter-approved-launch-template-id-here'
      				)
      			)
      			OR (
      				useragent not like 'autoscaling.amazonaws.com'
      				and eventname LIKE 'RunInstances'
      			)
      	) AS CloudTrail ON CostAndUsageReport.Instance_ID = CloudTrail.instance_Id
      
      

   4. You can now see the instances that are not deployed from the approved launch templates, alongside the users that created these instances and the cost incurred by the EC2:

Use Amazon Quicksight to visualize correlated data

In this section, you will visualize daily spend incurred by all the EC2 instances and separate them as per instances launched with approved launch template versus instances launched without approved template. This visualization will help you understand the trend in your environment. To do so, you will use QuickSight that visualizes a direct query in Athena. For the query, you will correlate data from CloudTrail with CUR data.

1. 1. Navigate to your Amazon Quicksight home page and select Analyses from left panel.
   2. From the right panel, choose New analysis and then select New dataset. Choose the Athena data source card.
   3. Provide a Data Source name and select Athena workgroup as “Primary”.
   4. Choose Validate connection and then choose Create data source. Choose Catalog as “AwsDataCatalog”.
   5. Select Database as “default” and choose Use custom SQL. Enter the query below:

SELECT CloudTrail.Approval_STATUS,
	CostAndUsageReport.Instance_ID,
	CloudTrail.User_ARN,
	CostAndUsageReport.Unblended_COST,
	CostAndUsageReport.Description_COST,
	CostAndUsageReport.Start_DATE,
	CostAndUsageReport.End_DATE
FROM (
		SELECT line_item_resource_id as Instance_ID,
			line_item_product_code,
			line_item_line_item_description as Description_COST,
			line_item_usage_start_date as Start_DATE,
			line_item_usage_end_date as End_DATE,
			SUM(line_item_unblended_cost) AS Unblended_COST
		FROM "enter-your-cur-database"."enter-your-cur-table"
		WHERE line_item_product_code = 'AmazonEC2'
			AND line_item_line_item_type NOT IN ('Tax', 'Refund', 'Credit')
		GROUP BY line_item_resource_id,
			line_item_product_code,
			line_item_usage_start_date,
			line_item_usage_end_date,
			line_item_line_item_description
		ORDER BY Unblended_COST DESC
	) AS CostAndUsageReport
	JOIN (
		SELECT useridentity.arn as User_ARN,
			json_extract_scalar(
				element_at(
					CAST(responseelements AS map(varchar, varchar)),
					'instancesSet'
				),
				'$.items[0].instanceId'
			) as instance_id,
			Case
				WHEN (
					useragent like 'autoscaling.amazonaws.com'
					and NOT(
						json_extract_scalar(
							element_at(requestparameters, 'launchTemplate'),
							'$.launchTemplateId'
						) like 'enter-approved-launch-template-id-here'
					)
				)
				OR (
					useragent not like 'autoscaling.amazonaws.com'
					and eventname LIKE 'RunInstances'
				) Then 'Non Approved' Else 'Approved Templates'
			End as Approval_STATUS
		FROM "aws:cloudtrail"."enter-eds-id-here"
	) AS CloudTrail ON CostAndUsageReport.Instance_ID = CloudTrail.instance_Id

6.  You can then Publish & Visualize the data. To visualize, in Visuals, select Vertical bar chart under Change Visual Type. For X AXIS, choose Start_DATE, for VALUE choose Unblended_COST, for GROUP/COLOR choose Approval_STATUS. To learn more about formatting in QuickSight, please refer to the  formatting a visual documentation.

Cleaning up

To avoid incurring future charges, delete the resources you created in this blog.

1. Delete IAM role and IAM policy created in Step 1.
2. Delete the event data store created in Step 2.
3. Delete any datasets from QuickSight that are not needed.
4. Delete any analysis from QuickSight that is not needed.

Conclusion

In this blog post, you learned how you can use AWS CloudTrail Lake in order to audit, report, and visualize correlated data for ephemeral EC2 instances. CloudTrail Lake as a zero-ETL data source for Amazon Athena now enables you to do this without moving data or managing pipelines. Using the AWS Config AWS::AutoScaling::AutoScalingGroup configuration item you can audit and track the definition of your instances via the launch template defined in the Auto Scaling group. This provides evidence of how the instances are configured at launch and whether the instances were created from an approved template which you can confirm by querying our CloudTrail Lake logs from Athena. Each time an instance is launched we track API activity in CloudTrail Lake which provides evidence of when an instance was created or terminated. Any instances that are not launched from an approved template can then be visualized in Amazon QuickSight for analysis or remediation. For more content on how you can leverage AWS CloudTrail lake see the posts below:

View multi-account Service Quotas Increase using AWS CloudTrail Lake
Find the most evaluated AWS Config rules using AWS CloudTrail Lake
AWS CloudTrail Lake Supports Ingesting Activity Events From Non-AWS Sources
Investigate security events by using AWS CloudTrail Lake advanced queries

About the authors