AWS Marketplace

Automate multi account observability in AWS using Cribl and AWS Control Tower

Having a multi-account strategy is a best practice to achieve higher isolation of resources. It also helps to meet regulatory and compliance needs, track operational costs, and add an extra layer of security. AWS Control Tower uses AWS best practices to establish a well-architected, multi-account baseline and also enables governance across your AWS accounts. Many customers use AWS Control Tower to manage and govern multi-account AWS environments. For more information about managing multi-account AWS environments with AWS Control Tower, see Getting Started with AWS Control Tower.

Cribl is a member of the AWS Partner Network (APN) as an ISV Technology Partner. Cribl LogStream is a vendor-neutral, purpose-built streaming pipeline for logs, metrics, and observability data. LogStream helps you implement an observability pipeline, which allows you to route machine data from any source to any destination while transforming your data in motion. This process enriches the data with additional context and secures its contents. LogStream enables you to parse, restructure, and enrich data in flight. This helps you to get the right data, where you want, in the format you desire.

In this post, Kamilo and I share a new solution that integrates Cribl with AWS Control Tower. Using this solution, all newly added AWS accounts in an AWS Control Tower environment can be automatically enrolled with Cribl using Account Factory. The integration facilitates Cribl-based monitoring and analytics to be automatically enabled for all newly added AWS accounts.

Prerequisites

You must complete the following prerequisites before implementing the Cribl and AWS Control Tower integration solution:

  1. Subscribe to Cribl LogStream via AWS Marketplace. In the upper right, choose Continue to Subscribe.
  2. After successfully subscribing to Cribl Logstream, choose Accept Terms.

Solution Overview

The AWS Control Tower integration with Cribl is based on the automation of AWS Control Tower lifecycle events via Amazon CloudWatch events and AWS CloudFormation StackSets. The integration consists of one AWS CloudFormation template that fully automates the provisioning, setup, and integration of all the components necessary for this solution.

The AWS CloudFormation template and a detailed README for this solution is available here. This template is deployed in the AWS Control Tower management account and creates the following components:

  1. A Cribl AWS CloudFormation StackSet in the AWS Control Tower management account. This incorporates the following Cribl components for setting up a Cribl LogStream single instance:
    1. An Amazon CloudWatch Events rule: Triggered based on an AWS Control Tower lifecycle event.
    2. An AWS Lambda lifecycle function: The target for the CloudWatch Events rule.
  2. A Cribl AWS CloudFormation stack instance in the AWS Control Tower managed account.
    1. When a new account is added from the AWS Control Tower management account, the Lambda function creates a stack instance in the managed account. The stack instance is based on the Cribl StackSet deployed in the management account and provisions the single instance in the managed account

Solution architecture

The following architecture diagram illustrates the components and flow of the AWS Control Tower and the Cribl integration.

  • In the AWS Control Tower management account:
    • In the AWS Service Catalog via the Account Factory, an administrator provisions a new AWS Control Tower account.
    • After the administrator successfully provisions a new account, an AWS Control Tower lifecycle event triggers a CloudWatch Events rule.
    • The AWS CloudWatch Events rule triggers a Lambda function.
    • An AWS CloudFormation StackSet launches the Cribl stack instance in the AWS Control Tower managed account.
  • In the AWS Control Tower managed account:
    • The Cribl template that provisions the Cribl LogStream single instance in the managed account.

Refer to the following architecture diagram:

The diagram describes the solution architecture and interactions between the various components of the AWS Control Tower and Cribl integration

Step-by-step walkthrough

Follow these steps to set up the Cribl integration with AWS Control Tower.

A. Set up Cribl integration with AWS Control Tower

Perform the following steps to set up Cribl integration with AWS Control Tower:

  1. Log in to the AWS CloudFormation console of your management account.
  2. Launch the aws-cribl-controltower.yaml template from the AWS CloudFormation console. To launch a AWS CloudFormation template from the console, follow the steps for starting the Create Stack wizard.

B. Test your integration

Perform the following to test your integration by adding a managed account and creating a lifecycle event.

Add the managed account

Perform the following steps to add the managed account:

  • Log in to the AWS Control Tower management account and navigate to the AWS Control Tower console.
  • To enroll a new managed account in the AWS Control Tower organization, in the navigation pane, choose Account Factory.
  • Enter values for Account email, Display name, AWS SSO email, AWS SSO user name, and Organizational unit.
  • Choose Enroll account.

It can take up to 30 minutes for the account to be created and the AWS Control Tower lifecycle event to trigger.

Test your integration

To check that the integration is working, perform the following steps:

  1. Log in to the AWS Control Tower managed account. Navigate to the CloudFormation console. Check that there is an AWS CloudFormation stack instance in this account that launches the Cribl LogStream single instance. From the navigation pane, select this stack instance and choose Stack info. The status field should display a value of CREATE_COMPLETE.
  2. With the stack instance selected in step 2a, select the Resources tab and look for the instanceip field. Make a note of the value of this field. This is the Public IP Address and the instanceip of this single instance.
  3. Using the instanceip obtained in step 2b, to validate that the Logstream instance is running, you can either navigate to http://<instanceip>:9000/dashboard/stats or select the Monitoring link located in the navigation bar.

Conclusion

In this post, we showed you how to automatically enroll new AWS Control Tower accounts with Cribl. Cribl’s integration with AWS Control Tower enables you to automatically extend Cribl-based monitoring and analytics to be automatically enabled for all newly added AWS accounts. For more information about this solution, see Solutions for AWS Control Tower in AWS Marketplace.

About the Authors

Kanishk Mahajan is an ISV Solutions Architecture Lead at AWS. In this role, he leads cloud transformation and solution architecture for ISV partners and mutual customers. Kanishk specializes in management and governance, migrations and modernizations, and security and compliance. He is a Technical Field Community (TFC) member in each of those domains at AWS.

Kam Amir

Kamilo “Kam” Amir is the Director of Business Development for Cribl and is based in the Washington, D.C. area. He’s been with Cribl since LogStream version 2.2 and leads the technical alliances program. If you need to find him, just look for him hiking in Rock Creek Park with his family and husky or in the Cribl Slack Community.