Announcing Live Tail feature for Amazon CloudWatch Logs

Think about the following scenarios:

Do you Secure Shell (SSH) into multiple application servers, or use tail -f command from the terminal sessions and begin tailing log files to debug errors, or validate if a process has started correctly, or verify application/service configuration changes, or monitor deployments/service-impacting changes, etc.?
Besides custom application logs, do you wish to tail logs vended by 50+ AWS services such as Amazon Virtual Private Cloud (VPC), Amazon Route 53, AWS Lambda, Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), etc. to CloudWatch Logs and filter and/or highlight across these diverse set of logs for error, exception, or any other keywords of interest?
Do you export logs from CloudWatch to custom or third-party monitoring software to tail logs data and troubleshoot?

If you have answered yes to one or all of the above questions, we’re with you. Having a rich visual analytics experience of application logs in real-time can aid operational troubleshooting, investigations, and root cause analysis. Amazon CloudWatch Logs has launched Live Tail, a new interactive analytics capability that provides you a real-time view of your incoming CloudWatch logs data.

In this blog post, we will show how to use the interactive Live Tail capability in CloudWatch Logs and debug application availability errors.

Let’s get started.

Prerequisites

To start a Live Tail session, you should either assume the AWS Identity and Access Management (IAM) Admin Role or have an IAM policy allowing logs:* or add logs:StartLiveTail and logs:StopLiveTail actions to your policy statement. Alternatively, you could add CloudWatchLogsReadOnlyAccess managed IAM policy to grant necessary permissions to use the AWS console to start and stop CloudWatch Logs Live Tail sessions and avoid having to investigate what permissions are needed.

The following is an example of a permissions policy. Refer to Using identity-based policies (IAM policies) for CloudWatch Logs for examples of IAM identity-based policies for controlling access to CloudWatch Logs.

{
  "Version": "2012-10-17",
  "Statement": [
   {
   "Effect": "Allow",
   "Action": [
  "logs:StartLiveTail",
"logs:StopLiveTail",
"logs:Get*"
   ],
   "Resource": [
   "arn:aws:logs:*:*:*"
   ]
  }
]
}

Getting started with Live Tail from the console

You can start a Live Tail session through one of the following methods.

Method 1

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
In the navigation pane, choose Logs, Live tail.
For Select log groups, select the log group(s) that you want to view events from, in the Live Tail session. You can select as many as 10 log groups.
(Optional) If you selected only one log group, you can filter your Live Tail session further by selecting one or more log streams to view log events from. To do so, under Select log streams, select the names of the log streams from the dropdown list. Alternatively, you can use the second box under Select log streams to enter a log stream name prefix, and then all log streams with names that match the prefix will be selected. Also, you can add filter pattern(s).

Method 2

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
In the navigation pane, choose Logs, and then choose Log groups.
Choose one or more log groups. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account.
You will see “Start tailing” button becoming available.

Method 3

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
In the navigation pane, choose Logs, and then choose Logs Insights.
In the Select log group(s) drop down, choose one or more log groups. If this is a monitoring account in CloudWatch cross-account observability, you can select log groups in the source accounts as well as the monitoring account.
You will see “Start tailing” button becoming available.

Feature walkthrough

Once you launch Live Tail from the CloudWatch console, as shown in figure 1, a left-side pop-out filter panel displays a mandatory field for you to select log group(s) across multiple accounts, plus optional fields to select log stream(s) and filtering pattern(s).

xx events/s, % displayed on the top right of the console shows the event display stats. Event display on the console is controlled at a certain number of events per second. CloudWatch Logs attempts to display as many log events as possible on the console to prevent from omitting events that are important for your purpose. When volume of log events is very high, ‘% displayed’ shows the percentage of ingested log events that are displayed. You can use the filters to narrow down your search to the events you are interested in.

Events above the maximum rate are not displayed (i.e. % displayed) on the console, but continue their journey to be processed and stored in CloudWatch Logs. You also see a timer on the top right of the console to show the current Live Tail session’s duration.

Figure 1 — Live Tail console

You can specify filter pattern to display only log events that contain certain words or other strings (i.e., ERROR, exception, failure) and cut out the noise. The filters field is case-sensitive. Refer to Using filter patterns to match terms in log events to learn more about how to use filter patterns to match terms in log events. Terms can be words, exact phrases, or numeric values.

Additionally, highlighting is a powerful feature that you can use to specify up to 5 terms to mark each event with a color-coded indicator(s) as shown in figure 2. For example, assume you have filtered on events containing the ‘ERROR’ keyword, but there are many error types that you want to be aware of. Let’s say that you want to quickly identify the filtered events containing 404 error type. In this case, you simply add 404 as a highlighting term and each event will be marked with the assigned color indicator.

Once you identify the desired event in the events window,

Select the magnifying glass next to any event, the events flow will pause and a right-side panel will pop out to display the event’s details and other related options.
If you click on View trailing events option in the side panel, the Log Groups console will launch in another browser tab displaying the selected event with the surrounding events. This is helpful to expand the view to leading/trailing events.
Scroll to latest events button becomes available on the lower right corner of the console to take you to the latest live events immediately.
Your Live Tail session will be stopped if there are no actions on the console page and you can choose Restart this session button to resume the session.

Figure 2 — Live Tail console showing expanded side panel on event selection

Conclusion

In this post, you learned how to tail the logs right from within CloudWatch Logs and debug application availability errors. If you have questions or feedback, then open an AWS Support Case.