Containers
Centralized logging for Windows containers on Amazon ECS using Fluent Bit
Introduction
Today, Amazon Web Services (AWS) announced the support for Fluent Bit container images for Windows operating system. This support eliminates the need for Windows customers to implement any custom logging solutions in their application code or manage custom agents on their Windows instances to scrape the logs. For more details about the supported Windows versions and the image tags for Fluent Bit, please visit our documentation here.
Fluent Bit is a fast and flexible log processor and router supported by various operating systems. It’s used to route logs to various AWS destinations, such as Amazon CloudWatch, Amazon Kinesis Data Firehose destinations, Amazon Simple Storage Service (Amazon S3), and Amazon OpenSearch. In addition to the AWS destinations, Fluent Bit supports common partner solutions such as Datadog, Splunk, custom HTTP servers, and many more. For more details about container log routing, please visit the blog link here. AWS already supports the Fluent Bit container images based on Linux operating system. With today’s release on Windows support, it now allows customers to have a centralized mechanism for processing and routing their logs across Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) for both Linux and Windows workloads.
In this post, we cover how Amazon ECS customers can deploy Fluent Bit containers on their Windows instances running in Amazon ECS to stream Internet Information Services (IIS) logs generated by the Windows tasks to Amazon CloudWatch logs to centralize logging. You can also use Fluent Bit on Amazon EKS and create a unified logging experience across Amazon ECS and Amazon EKS. For more information about deploying Fluent Bit on Amazon EKS, visit our blog post here.
Architecture overview
Architecture of setup used in this post
In order to set up Fluent Bit on Amazon ECS, we’ll use the following approach:
- Run Fluent Bit as a standalone container on the instance such that:
- It listens on port
24224
using the forward input plug-in. - We expose the port
24224
to the host so that docker runtime send logs to Fluent Bit using this exposed port. - Fluent Bit has its own configuration that enables it to send the logs records to specified destinations.
- For production usage, we suggest running Fluent Bit task as an Amazon ECS Service with Daemon scheduling strategy. This ensures that a single instance of Fluent Bit always runs on your container instances in the cluster. For step-by-step guide, visit our Amazon ECS tutorial.
- Fluent Bit task should be started prior to your application tasks so that it can capture all the application logs.
- It listens on port
- All other Amazon ECS task containers are launched using the
fluentd
docker logging driver.- Docker connects to the TCP socket
24224
onlocalhost
inside the host namespace. - Amazon ECS agent adds labels to the containers, which include cluster name, task definition family name, task definition revision number, task ARN, and the container name. We’ll add the same to the log record using labels option of the
fluentd
docker logging driver. - Because we are setting
async
option of thefluentd
logging driver as true, even if the Fluent Bit container is restarted, docker buffers the logs until Fluent Bit container is restarted. You can increase the buffer limit by setting the fluentd-buffer-limit option.
- Docker connects to the TCP socket
Therefore, the end-to-end workflow is as follows:
- Fluent Bit container is started and listens on port
24224
, which is exposed to the host. - Fluent Bit uses the Task AWS Identity and Access Management (AWS IAM) role credentials as specified in its task definition.
- Other Amazon ECS tasks are launched on the same instance, which use
fluentd
docker logging driver to connect to the Fluent Bit container on port24224
. - When logs are generated by application containers, docker runtime tags those records, adds additional metadata specified in labels, and forwards them on port
24224
in the host namespace. - Fluent Bit receives the log record on port
24224
since it’s exposed to the host namespace. - Fluent Bit perform its internal processing and routes the logs as specified.
We have created some default Fluent Bit configurations to help you get started with minimal effort. For more information, visit the documentation here.
In this post, we’ll use the default Amazon CloudWatch Fluent Bit configuration, which would:
- Create a new log group for each Amazon ECS cluster and Amazon ECS task definition family.
- Create a new log stream for each task container in above generated log group whenever a new task is launched. Each stream is marked with the
task-id
to which the container belongs. - Add additional metadata such as Amazon ECS cluster name, Amazon ECS task ARN, Amazon ECS task container name, Amazon ECS task definition family, and the Amazon ECS task definition revision number in each log entry.
For example, if you have the following task definitions with the containers as:
Assuming each of these task definitions has a single task running in an Amazon ECS cluster named windows
, then we get the following Amazon CloudWatch log groups and streams:
/aws/ecs/windows.ecs_task_1
task-out.<TASK_ID>.container_1
task-out.<TASK_ID>.container_2
/aws/ecs/windows.ecs_task_2
task-out.<TASK_ID>.container_3
Prerequisites
Prerequisites and assumptions:
- The latest version of the AWS Command Line Interface (AWS CLI) is installed and configured. For more information, see Installing the AWS Command Line Interface.
- The
aws-for-fluent-bit
container image is available for the following Windows operating systems and therefore, you should be using the same:- Windows Server 2019 Core
- Windows Server 2019 Full
- Windows Server 2022 Core
- Windows Server 2022 Full
- The steps in Set up to use Amazon ECS have been completed.
- You have a cluster. In this post, the cluster name is
FluentBit-cluster.
For more information about creating Amazon ECS cluster, visit our docs here. - A Virtual Private Cloud (VPC) with a public subnet where the Amazon EC2 instance is launched. You can use your default VPC. You can also use a private subnet provided that Amazon CloudWatch endpoints are reachable from inside the private subnet. For more information regarding VPC, visit our documentation here.
- In order to build a Windows container image, you need to have created an Amazon Elastic Compute Cloud (Amazon EC2) Windows instance with Docker installed, which is based on the same Windows version as the Amazon ECS container instance. Alternatively, you can also use Amazon EC2 Image Builder for building your container image.
- An Amazon ECR repository to store the Windows container images. For more information about creating the same, visit our documentation here.
- Amazon ECS Exec prerequisites are completed if you wish to access your task containers to generate the logs. For more information, visit our documentation here.
Solution overview
In this post, we’ll complete the following tasks:
- Create the IAM access roles
- [Optional] Build a Windows application container image containing IIS and LogMonitor
- Create an Amazon ECS Windows container instance
- Configuration used for Fluent Bit
- Register a Windows Fluent Bit task definition which routes the logs to Amazon CloudWatch
- Run the above registered
ecs-windows-fluent-bit
task definition - Register a Windows task definition with application containers which generates the logs
- Run the above registered
windows-iis-task
task definition - [Optional] Access the IIS pods to generate logs
- Verify the logs on Amazon CloudWatch
- Cleaning up
Walkthrough
1. Create the AWS IAM access roles
Create the required AWS IAM roles.
1. Create a new role named buildInstanceRole
, which is used by your Windows build instance to create and push Amazon ECR images. If you are running in a production environment, then we recommend that you customize the role policies for minimal permissions.
For this post, you need to push docker image to Amazon ECR. Therefore, attach the following inline policy to your build instance IAM role.
To attach the inline policy to your instance IAM role, perform the following operations:
- Save the above content in a file named
ecr-push-policy.json
. - Run the following command to attach the inline policy to above created
buildInstanceRole
IAM role.
aws iam put-role-policy --role-name buildInstanceRole --policy-name ecr-push --policy-document file://ecr-push-policy.json
2. Create the Amazon ECS container instance role named ecsInstanceRole
. For more information, see Amazon ECS container instance IAM role.
3. Create an IAM role for the Fluent Bit task named fluentTaskRole
. For more information, see Creating an IAM role and policy for your tasks.
The AWS IAM permissions granted in this IAM role are assumed by the task containers. In order to allow Fluent Bit to send logs to Amazon CloudWatch, you need to attach the following permissions to the task AWS IAM role.
To attach the inline policy to your task AWS IAM role:
- Save the above content in a file named
fluent-bit-policy.json
. - Run the following command to attach the inline policy to
fluentTaskRole
AWS IAM role.
aws iam put-role-policy --role-name fluentTaskRole --policy-name fluent-bit-policy --policy-document file://fluent-bit-policy.json
4. [Optional] Create an AWS IAM role for the Windows task named iisTaskRole
, which is used to generate the logs. This is required only if you want you access your Windows task containers for generating logs as described in Step 9.
In order to turn on Amazon ECS Exec for your task, you need to attach the following permissions to the task IAM role. For more information about Amazon ECS Exec, see Using Amazon ECS Exec for debugging in Amazon ECS guide.
To attach the inline policy to your task AWS IAM role:
- Save the above content in a file named
ecs-exec-policy.json
. - Run the following command to attach the inline policy to
iisTaskRole
IAM role.
aws iam put-role-policy --role-name iisTaskRole --policy-name ecs-exec-policy --policy-document file://ecs-exec-policy.json
2. [Optional] Build a Windows application container image containing IIS and LogMonitor
If you already have a Windows container image for your application, you can skip this section.
2.1 To test the functionality explained in this post, create a Windows container image containing IIS and LogMonitor. By default, Windows containers send logs to Event Tracing for Windows (ETW), Event Log, and custom logs files. However, log processors such as Fluent Bit and fluentd
fetch containers logs from a STDOUT
pipeline which doesn’t exist on Windows containers. LogMonitor is an open-source plugin created by Microsoft to create a STDOUT
pipeline within the Windows container so such tools can successfully fetch logs in the same way they do on a Linux environment.
For more instructions about how to use LogMonitor, see the GitHub repository or Microsoft post.
In the example below, we have a Dockerfile to build the Windows container image containing IIS and LogMonitor.
This sample LogMonitorConfig
configuration retrieves all the logs files with the extension .log saved in C:\inetpub\logs
and sub directories, including the IIS access logs.
As the build completes, push the image to your Amazon ECR repository.
3. Create an Amazon ECS Windows container instance
Create an Amazon ECS Windows container instance.
To create an Amazon ECS instance;
- Use the
aws ssm get-parameters
command to retrieve the AMI ID for the Region that hosts your VPC. For more information, see Retrieving Amazon ECS-Optimized AMI metadata. - Use the Amazon EC2 console to launch the instance.
- Open the Amazon EC2 Console at https://console.aws.amazon.com/ec2/.
- From the navigation bar, select the Region to use.
- From the EC2 Dashboard, choose Launch instance.
- For Name, enter a unique name.
- For Application and OS Images (Amazon Machine Image), choose the AMI that you retrieved in the first step.
- For Instance type, choose t3.xlarge.
- For Key pair (login), choose a key pair.
- Under Network settings, for Security group, choose an existing security group, or create a new one.
- Under Network settings, for Auto-assign Public IP, select Enable.
- Under Advanced details, for IAM instance profile, choose
ecsInstanceRole
. - Configure your Amazon ECS container instance with the following user data. Under Advanced Details, paste the following script into the User data field, replacing
cluster_name
with the name of your cluster (FluentBit-cluster
in this tutorial). - When you are ready, select the acknowledgment field, and then choose Launch Instances.
- A confirmation page lets you know that your instance is launching. Choose View Instances to close the confirmation page and return to the console.
4. Configuration used for Fluent Bit
To ensure that the Windows customers can get started quickly, we have crafted default configurations for the following Fluent Bit output destinations which are present in the container images:
In order to use the same, you need to ensure that:
- Your task AWS IAM role has sufficient permissions as mentioned in the Fluent Bit output plugin documentation.
- You have set the environment variables required for the specific configuration as documented here.
- You have overridden the
entrypoint
of the Windows Fluent Bit image with the specific configuration which you want to use. - You have configured your Amazon ECS task definition for your application tasks to use
fluentd
logging driver and the following labels are configured in labels option offluentd
docker logging driver:amazonaws.ecs.cluster
amazonaws.ecs.container-name
amazonaws.ecs.task-arn
amazonaws.ecs.task-definition-family
amazonaws.ecs.task-definition-version
These labels are added by Amazon ECS agent to each container.
In this post, we’ll show the above steps for Amazon CloudWatch default configuration but it can be changed easily to other destinations following similar mechanism. For more details about each default configuration, please visit our documentation here.
The default Amazon CloudWatch Windows Fluent Bit configuration is:
Every log that gets into Fluent Bit has a tag that you specify, or it is automatically generated when you do not supply one. The tags can be used for routing the log entry to different filters and output plugins. For additional information, see Tag in the Fluent Bit Official Manual.
In this Fluent Bit configuration:
- We have configured forward input plugin such that it listens for incoming traffic on TCP port
24224
. Each log entry received on that port would have a tag which would be modified by this plugin to prefix the same withecs.
string. - As a next step in Fluent Bit internal pipeline, the log entry would be routed to
modify
filter using the Match regex. This filter replaces the keys within log record json to the format which can be consumed by Fluent Bit. - The modified log entry is consumed by the
rewrite_tag
filter. This filter changes the tag of the log record to the formatout.<TASK_ID>.<CONTAINER_NAME>
. - The new tag is routed to output
cloudwatch_logs
plugin, which creates the log groups and streams as described earlier in the post. This is accomplished usinglog_group_template
andlog_stream_prefix
options of the Amazon CloudWatch output plugin.
5. Register a Windows Fluent Bit task definition which routes the logs to Amazon CloudWatch
Register a Windows Fluent Bit task definition which routes the logs to Amazon CloudWatch.
To register a task definition:
- Create a file named
fluent-bit.json
with the following contents.
Replace the following variables:
task-iam-role
with the Amazon Resource Name (ARN) of your task AWS IAM role (fluentTaskRole
created in step 1).region
with the Region where your task runs.
This task definition overrides the default entrypoint with a custom entrypoint command:
Powershell -Command C:\\entrypoint.ps1 -ConfigFile C:\\ecs_windows_forward_daemon\\cloudwatch.conf
Additionally, as documented in the configuration here, we have added the environment variable AWS_REGION
to the container definition.
Note: This task definition exposes Fluent Bit container port 24224 to the host port 24224. Verify that this port is not open in your Amazon EC2 instance security group to prevent access from outside. For more information about security groups, refer to Amazon VPC docs here.
- Run the following command to register the task definition.
Replace the following variables:
-
region
with the Region where your task definition needs to be registered
aws ecs register-task-definition --cli-input-json file://fluent-bit.json --region region
You can list the task definitions for your account by running the list-task-definitions
command. The output of displays the family and revision values that you can use together with run-task
or start-task
.
6. Run the registered ecs-windows-fluent-bit task definition
After you register a task definition for your account, you can run a task in the cluster. For this tutorial, you run one instance of the ecs-windows-fluent-bit:1
task definition in your FluentBit-cluster
cluster. For production usage, you can consider running the ecs-windows-fluent-bit:1
task definition using Amazon ECS Service with Daemon scheduling strategy. This ensures that a single instance of Fluent Bit always runs on your container instances in the cluster.
To run a task
1. Run the following command to run the ecs-windows-fluent-bit:1
task definition that you registered in the previous step.
Replace the following variables:
region
with the Region where your task runs.
aws ecs run-task --cluster FluentBit-cluster --task-definition ecs-windows-fluent-bit:1 --region region
Note: Because ecs-windows-fluent-bit
uses awslogs
logging driver, your container instance needs to have the necessary permissions.
2. Run the following command to list your tasks.
Replace the following variables:
region
with the Region where your task runs.
aws ecs list-tasks --cluster FluentBit-cluster --region region
7. Register a Windows task definition with application containers which generates the logs
Register a task definition which generates the logs. This task definition deploys Windows container image containing IIS and LogMonitor which we had created in Step 3.
To register a task definition
- Create a file named
windows-iis-task.json
with the following contents.
Replace the following variables:
- [Optional]
iis-task-iam-role
with the ARN of your IIS task AWS IAM role (iisTaskRole
created in step 1). iis_container_image
with the Windows container image created in Step 2.
Note: This task uses the default
network mode; however, you can also use the awsvpc
network mode with the task.
In this task definition, we configure the container to use fluentd docker logging driver. The fluentd
logging driver connects to port 24224
on which Fluent Bit forward input plugin is listening. Amazon ECS agent labels each Amazon ECS container with tags, which include the cluster name, task ARN, task definition family name, task definition revision number, and the task container name. We’ll pass these key-value labels
to Fluent Bit.
- Run the following command to register the task definition.
Replace the following variables:
region
with the Region where your task runs
aws ecs register-task-definition --cli-input-json file://windows-iis-task.json --region region
You can list the task definitions for your account by running the list-task-definitions command. The output of displays the family and revision values that you can use together with run-task
or start-task
.
8. Run the registered windows-iis-task task definition
After you register the windows-iis-task
task definition, run multiple counts of the same in your FluentBit-cluster
cluster.
To run a task
- Run the
windows-iis-task:1
task definition you registered in the previous step.
Replace the following variables:
region
with the Region where your task runs
aws ecs run-task --cluster FluentBit-cluster --task-definition windows-iis-task:1 --count 2 --enable-execute-command --region region
If you don’t want to access task containers using Amazon ECS Exec, then omit --enable-execute-command
from above command.
- Run the following command to list your tasks.
Replace the following variables:
region
with the Region where your task runs.
aws ecs list-tasks --cluster FluentBit-cluster --region region
9. [Optional] Access the IIS pods to generate logs
You can either wait for actual traffic to hit your Windows pod hosting the IIS webserver, or you can log into your container and force the logs.
9 Run the following command to log into your container-:
Replace the following variables:
task_id
with the task id of any of thewindows-iis-task
started in Step 8.region
with the Region where your task runs.
aws ecs execute-command --cluster FluentBit-cluster --task task_id --container sample-container --interactive --region region --command powershell.exe
9 From inside the container, run the following command to hit the web server:
Invoke-WebRequest localhost -UseBasicParsing
10. Verify the logs on Amazon CloudWatch
In order to verify your Fluent Bit setup, check for the following log groups in the Amazon CloudWatch Console:
/ecs/fluent-bit-logs
– This contains the log stream corresponding to the Fluent Bit daemon container which is running on the container instance./aws/ecs/FluentBit-cluster.windows-iis-task
– This is the log group which corresponds to all the tasks launched forwindows-iis-task
task definition family insideFluentBit-cluster
Amazon ECS cluster.task-out.<FIRST_TASK_ID>.sample-container
– This log stream contains all the logs generated by sample-container task container.task-out.<SECOND_TASK_ID>.sample-container
– This log stream contains all the logs generated by sample-container task container.
Inside the task-out.<TASK_ID>.sample-container
log stream, each log entry would have the following fields:
If you see these log groups and streams in the Amazon CloudWatch console, then Fluent Bit setup is verified.
Cleaning up
When you have finished, clean up the resources associated with it to avoid incurring charges for resources that you aren’t using.
To clean up the tutorial resources
- Stop the
windows-iis-task
task and theecs-windows-fluent-bit
task. For more information, see Stopping tasks using the new console. - Run the following command to delete the log groups created in this post. For more information about deleting log groups see delete-log-group in the AWS Command Line Interface Reference.
aws logs delete-log-group --log-group-name /ecs/fluent-bit-logs aws logs delete-log-group --log-group-name /aws/ecs/FluentBit-cluster.windows-iis-task
- Run the following command to terminate the Amazon EC2 instance created in this tutorial. Replace
INSTANCE_ID
with the id of Amazon EC2 instance created in Step 3.aws ec2 terminate-instances --instance-ids <INSTANCE_ID>
- Run the following command to delete the private Amazon ECR repository. Replace
ECR_REPOSITORY_NAME
with the Amazon ECR repository used in Step 2.aws ecr delete-repository --repository-name <ECR_REPOSITORY_NAME> --force
- Run the following command to detach the inline AWS IAM policies and delete the IAM roles created in this tutorial.
aws iam delete-role-policy --role-name buildInstanceRole --policy-name ecr-push aws iam delete-role --role-name buildInstanceRole aws iam delete-role --role-name ecsInstanceRole aws iam delete-role-policy --role-name fluentTaskRole --policy-name fluent-bit-policy aws iam delete-role --role-name fluentTaskRole aws iam delete-role-policy --role-name iisTaskRole --policy-name ecs-exec-policy aws iam delete-role --role-name iisTaskRole
- Run the following command to delete the Amazon ECS cluster.
aws ecs delete-cluster --cluster FluentBit-cluster
Conclusion
In this post, we showed you how to deploy Fluent Bit on your Windows instances in Amazon ECS.
Using Fluent Bit as a log router is a beneficial way to centralize logging across Amazon ECS for both Linux and Windows workloads. Fluent Bit can send the logs to various destination solutions allowing flexibility for customers.
Amazon ECS engineers are actively working to improve the experience for running Fluent Bit on Amazon ECS. For running Fluent Bit as a Daemon Service, track the progress in this roadmap issue. For the Firelens feature of Amazon ECS, track the progress in this roadmap issue.