AWS Marketplace

Improve security monitoring within your AWS environment with Logz.io

All of your teams, whether responsible for many cloud environments or a single application, must be able to understand the health and security of their operations easily. Your teams often use metrics based on security operations outcomes to gain useful insights.

AWS and Logz.io make it easier to bring together and analyze your logs so you can generate metrics, know the status of your security posture, and gain insight over time. These activities are supported centrally when you provide an observability solution for consumption, storage, analysis, and presentation of security data for analysis.

In this post, Mark and I will show you how to use Logz.io to build a centralized observability strategy to enhance visibility into your AWS environment and increase your security posture. By using dedicated analysis and visualization, you can consolidate large volumes of data from distributed sources to get a centralized view into your cloud security posture.

Logz.io® Cloud SIEM (security information and event management) aggregates security logs and alerts across distributed environments to enable your team to investigate security incidents from a single platform. You can detect threats and investigate security incidents in AWS as soon as you begin providing logs.

Prerequisites

When you use Logz.io Cloud SIEM, your logs are cross-referenced against multiple Threat Intelligence feeds to flag malicious internet protocols (IPs), domain name services (DNS), and URLs. The software then identifies potential methods of attack. Logz.io Cloud SIEM can help your team speed threat detection and monitor the security of even the largest environments at scale.

Solution walk through: Improve security monitoring within your AWS environment with Logz.io

Step 1: Subscribe to Logz.io in AWS Marketplace

  1. Log in to your AWS Management Console. Then navigate to the Logz.io service offering by either using the search bar or following this link to Logz.io Cloud SIEM (Free Tier).
  2. On the Logz.io product detail page, read external reviews, review pricing, and see usage or support details. In the upper right, choose the orange button marked either Continue to Subscribe or View purchase options.
  3. To set up your contract, follow the subscription wizard. You can select the length of the agreement, auto-renewal, the amount cap for daily ingestion, and how long to configure data retention. You’ll see a few standard options for each. You can choose to contract on a monthly basis, annually, or biannually.
    Standard subscriptions are in increments of 5GB/day. You can also add a purchase order number if you have one.
  4. When ready, choose Create Contract. If you have already set up a contract, you see options to upgrade or modify the contract, as well as modify terms for anticipated renewals.
  5. To set up your Logz.io account in Logz.io’s own user interface, choose the Set up your account button. You will be directed to logz.io. If you already have an account, you can log in. Otherwise, create a new account. To do that, at the top right of the screen, choose Start Free Trial.
  6. You will be given options to log in via external accounts. Enter your basic information, including the email you want to use, and select the Region you want to be in. Then choose Submit.

Step 2. Collecting logs from popular AWS services

Many AWS services generate log data that Logz.io can help aggregate and consolidate for security event correlation. In this post, Mark and I will show how to ship the top four AWS services crucial to secure your environment.

AWS CloudTrail

To collect relevant data from this service, follow these step-by-step instructions.

  1. In the Logz.io interface, go to Logs. Under the Shippers tab, go to Send your data, then Security Sources. Choose Ship CloudTrail logs.
  2. Send your logs to an Amazon Simple Storage Service (Amazon S3) bucket. Logz.io fetches your CloudTrail logs from an S3 bucket. For help with setting up a new trail, see Overview for Creating a Trail from AWS.
  3. Add your S3 bucket information. To use the S3 fetcher, log into your Logz.io account, and then go to the CloudTrail log shipping page.
    1. On the Ship CloudTrail logs page, choose + Add a bucket. Select your preferred method of authentication, either an IAM role or access keys. The configuration wizard will open.
    2. Enter the S3 bucket name.
    3. Enter your Prefix. That is your CloudTrail path. See further details in the following steps.
    4. There is no Region selection box because it is not needed. Logz.io will pull data from all Regions in AWS for the specified bucket and account.
    5. Get the information from your CloudTrail AWS path. When creating the bucket, you must fill in two parameters, {BUCKET_NAME} and {PREFIX}. You can find them in your CloudTrail AWS path.
  4. Check Logz.io for your logs. Give your logs some time to get from your system to Logz.io’s, and then open Kibana. If you still don’t see your logs, see log shipping troubleshooting.

This page provides more information about shipping CloudTrail logs to Logz.io.

Amazon GuardDuty

You can ship GuardDuty logs two ways: either through manual AWS Lambda configuration or automated AWS CloudFormation deployment. To collect relevant data from this service, follow these step-by-step instructions to ship GuardDuty logs into Logz.io.

Following is a brief summary of the manual Lambda configuration steps:

  1. Create a new Amazon Kinesis data stream and configure CloudWatch events.
  2. Create a new IAM role and a new Lambda function.
  3. Download the Amazon Kinesis stream shipper and unload the zip file.
  4. Set environment variables, configure the function’s basic settings, and set the Kinesis event trigger.
  5. Check Logz.io for your logs.

Amazon VPC Flow

The S3 fetcher’s offset is determined by the name of the last file fetched. Use standard AWS naming conventions to determine the file name ordering and to avoid log duplication. Log files are saved to the specified Amazon S3 bucket using a folder structure based on the flow log’s ID, Region, creation date, and destination options.

The file name of a log file is based on the log flow ID, Region, and creation date and time. To ship your VPC flow logs to Logz.io, do the following:

  1. Send your logs to an S3 bucket. Logz.io fetches your VPC flow logs from an S3 bucket. VPC flow logs are not stored in S3 by default, so you must set up AWS to send your flow logs to S3. For help with this, see Publishing Flow Logs to Amazon S3 from AWS.
  2. Add a new S3 bucket using the dedicated Logz.io configuration wizard. Log into the app to use the dedicated io configuration wizard and add a new S3 bucket. Provide your S3 bucket name in the configuration wizard and save.
  3. Check Logz.io for your logs. Give your logs some time to get from your system to Logz.io’s, and then open Kibana. If you still don’t see your logs, see log shipping troubleshooting.

To collect relevant data from VPC flow logs, follow these more detailed step-by-step instructions to ship VPC flow logs into Logz.io.

Elastic Load Balancing

When you set Logz.io to fetch Elastic Load Balancing (ELB) logs, Logz.io will periodically read logs from the configured S3 bucket. ELB logs are useful for application usage intelligence and monitoring. This ELB integration is specifically designed to work with the destination bucket to which ELB writes its logs.

It is based on ELB’s naming convention and path structure. If you’re looking to ship ELB logs from a different bucket, use the S3 Bucket shipping method instead. To ship logs from ELB, do the following:

  1. Send your logs to an S3 bucket. Logz.io fetches your ELB logs from an S3 bucket. For help with this, see the following AWS docs: Access Logs for Your Application Load Balancer, Monitor Your Network Load Balancers, and Enable Access Logs for Your Classic Load Balancer.
  2. Add a new S3 bucket using the dedicated Logz.io configuration wizard. Log into the Logz.io app to use the dedicated io configuration wizard and add a new S3 bucket. Enter your S3 bucket name in the configuration wizard and save.
  3. Check Logz.io for your logs. Give your logs some time to get from your system to Logz.io’s, and then open Kibana. If you still don’t see your logs, see log shipping troubleshooting.

To collect relevant data from this service, follow these step-by-step instructions to ship ELB logs.

3. Enable Logz.io consolidated view

After following steps 1 and 2, you can now get access to a consolidated view of your logs in Logz.io, The Logz.io Cloud SIEM dashboard offers a high-level overview of the AWS Well-Architected Framework. The dashboard includes the Logz.io’s definition of risks and recommendations of the five pillars, taken directly from the AWS Well-Architected Tool, This can help regularly evaluate workloads, identify high-risk issues, and record improvements. The dashboard also tracks your progress across the entire environment to build a secure, high-performing, resilient, and efficient infrastructure for your applications and workloads.

The following screenshot shows a dashboard, which provides a bar chart of workload by environments, a high and medium risk count, a bar chart showing high and medium risk for each pillar, and a list of lens improvements with associated URLs. It also shows links to your Amazon Guard Duty, AWS CloudTrail, Amazon VPC flow logs, and ELB dashboards.

enabled logz.io consolidated view

Conclusion

In this post, Mark and I showed how to use Logz.io to consolidate your security-related logging information. We also showed how to subscribe to and deploy Logz.io, collect AWS telemetry data from AWS CloudTrail, Amazon CloudWatch, and Amazon VPC flow logs. This will enable customers to take a holistic view of all logs in one location, which can help prevent security challenges.

Next steps

You can initiate your AWS security dashboards using Logz.io Cloud SIEM and gain immediate visibility across your entire cloud environment to improve your organization security posture.

To try this for yourself, subscribe to Logz.io Cloud SIEM, available in AWS Marketplace.

To learn more about cloud-based security operations center (SOC), download the whitepaper A cloud-based security operations center (SOC) helps improve your security detection and response.

About the authors

Jonah KowallJonah Kowall is CTO of Logz.io, an open-source observability and security platform for modern DevOps and SecOps teams.

.

.

.

mark kriafMark Kriaf is an AWS Partner Solution Architect.