AWS Cloud Operations Blog

Get visibility into the activity logs for your workforce and customer identities

The customers we work with often use identity services like Okta to authenticate their workforce identity into the AWS environment, and customer identity service to build authentication into their business applications. In addition, we see customers using other partner solutions in security, cost management, cloud operations, and observability categories. We have heard that our customers want visibility into the activity logs from non-AWS sources along with AWS sources in a single data source.

Using AWS CloudTrail Lake, which is a managed data lake, you can aggregate, immutably store, and query events recorded by CloudTrail across your AWS Accounts and Regions. CloudTrail Lake now supports ingesting activity logs from non-AWS sources. This lets you collect activity logs from any source in your hybrid environments, such as in-house or Software-as-a-Service (SaaS) applications hosted on-premises or in the cloud, virtual machines (VMs), or containers. You can immutably store the events for your auditing and compliance needs, security investigations, and operational troubleshooting. Furthermore, the stored data can be analyzed using a standard SQL-based queries.

Amazon EventBridge ingests data from supported SaaS platforms and routes to AWS Service targets. EventBridge lets you easily connect to and steam data from your SaaS applications without writing any code. Several AWS partners like Okta are already integrated with EventBridge and can start streaming the data with minimum configuration steps.

In this post, you’ll learn to get started sending activity logs from partners like Okta into CloudTrail Lake. You’ll leverage Okta’s ability to Add an Amazon EventBridge log stream to get logs into AWS, transform them into the CloudTrail format, and ingest them into CloudTrail Lake. In addition, you’ll utilize common SQL like statements to query the data and derive the required insights.

A few examples of the insights include the count of events for each target or event name, as well as a consolidated view of partner activity along with Amazon API Gateway activities in your environment.

Prerequisites

Before configuring this solution, conduct the following tasks:

When you deploy this solution, you’re responsible for the charges associated with additional services used, in addition to the EventBridge cost. Additional services include SQS queue, AWS Lambda, and CloudTrail Lake ingestion and query cost.

The source code for this solution is available here.

Solution Overview

Using the AWS CloudFormation template provided in this post, you can enable the streaming of activity logs from Okta into CloudTrail Lake. You can achieve this by extending the current EventBridge integrations with Okta log stream.

Solution Architecture diagram

Solution Architecture diagram

Deploy the CloudFormation stack in the AWS Account where you configured the prerequisites.

The CloudFormation stack provided as part of this post deploys the following:

  1. Create an EventBridge rule on the partner Event Bus. The partner Event bus is created when you enable AWS EventBridge log stream on Okta.
  2. The rule receives all the events from Okta and sends them to the target. In this solution, the Amazon Simple Queue Service (Amazon SQS) queue is used as the target.
  3. The events are queued in Amazon SQS and processed by a Lambda function. The Lambda function conducts the following:

a. Convert the Okta event into an CloudTrail Lake event schema

b. Ingest the event into CloudTrail managed Event Data Store.

  1. The queue is configured with Dead Letter Queue (DLQ) to collect any events that have failed to get ingested into CloudTrail Lake.
  2. Once the events are collected in the managed event data store, you can query and derive the insights of the events using SQL like queries.

Walkthrough

Step-1: Launch the solution

Download the CloudFormation template from the Git repository and launch it in your AWS account to deploy all the components shown above in the solution overview section.

  • Clone the Git repository or download the files to your local workstation.
  • Log in to the AWS Account where you performed the above prerequisites.
  • Create a CloudFormation stack. Type in the values for the following Parameter keys.

PartnerEventBusName: Navigate to Amazon EventBridge, Event buses, and Custom event bus, and note down the Name for the Event bus created for Okta integration.
PartnerRuleName: Enter any name for the rule (default: partner_event_process_rule)
ChannelArn: ARN of the channel that you created as part of the prerequisites.
S3BucketLocation: Amazon Simple Storage Service (Amazon S3) bucket location where the Lambda function packages are stored. You can leave the default value, unless you downloaded the code and hosted it in your S3 buckets.

  • Wait for the Stack Status to change to CREATE_COMPLETE, and then proceed to the next step.

Step-2: Generate some events on Okta

  • Log out from both the AWS accounts and your Okta console. Alternatively, you could use a new browser to perform the next steps.
  • Log in to the Okta Console with a user-ID that doesn’t exist to simulate a failure event.
  • Log in with the correct credentials to generate some additional events as well.

Step-3a: Query the ingested non-AWS events in CloudTrail Lake

  • Navigate to the AWS CloudTrail console.
  • Choose Lake on the left pane, and Editor.
  • Copy and paste the following query in the editor window.
SELECT eventData.eventTime as eventTime,
eventData.eventSource as eventSource,
eventData.eventname as eventName,
element_at(eventData.userIdentity.details, 'alternateId') as user,
element_at(eventData.userIdentity.details, 'displayName') as displayName
FROM 074365db-2079-4c18-ab83-e753897b74fe
where eventData.eventSource='okta.system-log-events'
and eventTime > '2023-01-30 22:22:30.000'
and eventTime < '2023-01-31 22:23:12.000'
  • Choose the Event data store that you created as part of the prerequisites. It will automatically update the event data store value in the query.
  • Adjust the eventTime to time duration after this solution deployment. The output looks like below.
Query non-AWS events from CloudTrail Lake

Query non-AWS events from CloudTrail Lake

Step-3b: Query both AWS and non-AWS events in CloudTrail Lake

As the user gets authenticated through Okta ang login to AWS, you could overlay the Okta events with AWS events to understand the complete set of activity that occurred. You could do this by querying two different event data stores with in CloudTrail lake. You can use UNION to group together the events. Using the below sample query, you can overlay both Okta and AWS events:

  • Copy and paste below query in the editor window.
SELECT eventData.eventTime as eventTime, eventData.eventSource as eventSource, eventData.eventname as eventName,
element_at(eventData.userIdentity.details, 'alternateId') as user,
element_at(eventData.userIdentity.details, 'displayName') as displayName
FROM 074365db-2079-4c18-ab83-e753897b74fe
where eventData.eventSource='okta.system-log-events'
and eventTime > '2022-09-08 22:22:30.000' and eventTime < '2022-09-08 22:23:12.000'
UNION
select eventTime, eventSource, eventName, userIdentity.principalid as user, userIdentity.type as displayName
from 181660c8-d56d-4ec1-aa32-c1713fb3bca3
where eventSource = 'sts.amazonaws.com'
and eventTime > '2022-09-08 22:22:30.000' and eventTime < '2022-09-08 22:23:12.000'
  1.  Adjust the event data store and eventTime as per your environment. The output looks like below.
Query both AWS and non-AWS events together from CloudTrail Lake

Query both AWS and non-AWS events together from CloudTrail Lake

Clean up

If you’re experimenting and want to clean the resources created after the testing or decide not to go with this approach, then you can delete the CloudFormation stack that you created as part of Step-1. This clears all the resources created by this solution.

Conclusion

In this post, you learned how to leverage EventBridge to send the activity logs from partners like Okta to CloudTrail Lake. You also ran a query to list the non-AWS events and overlay them with AWS events from CloudTrail Lake. The queries are built using commonly used SQL like statements to derive additional insights.

About the authors:

Kishore Vinjam

Kishore Vinjam is a partner solutions architect focusing on AWS Service Catalog, AWS Control Tower, and AWS Marketplace. He is passionate about working in cloud technologies, working with customers, and building solutions for them. When not working, he likes to spend time with his family, hike, and play volleyball and ping-pong.

Kapil Patil

Kapil Patil is a Sr. Partner Architect at Okta. He is part of the Tech Strategy and Partnership team to identify and define technology partnerships with ISV partners for Okta.