Integration & Automation

Logging Windows Amazon EC2 UserData activity in Amazon CloudWatch

UserData is a set of instructions you send to an Amazon Elastic Compute Cloud (Amazon EC2) instance at creation time to customize the deployment. These are tasks that happen automatically when the instance is launching for the first time and tend to be difficult to troubleshoot.

The process requires decrypting the instance credentials, connecting to the instance and navigating to the file location, and reviewing the log. This sounds pretty simple, especially when you have to do it only a couple of times, but after that it can get pretty tedious.

In this post, I want to show you how to minimize or eliminate the need to connect to your Amazon EC2 instance by sending key event data to Amazon CloudWatch. This allows you to map out the initialization events in a simple and centralized location.

When launching an EC2 instance, you have the option to send instructions to the instance at launch time. This can be accomplished by using the Amazon Web Services (AWS) Management Console, the AWS Command Line Interface, or AWS CloudFormation. Because this blog is geared toward Windows, we will be provisioning our instance using PowerShell.

In Part 1 of this post, I break down the logging function. Part 2 shows how to use the function. Part 3 shows a PowerShell EC2 launch script that combines what we cover in Parts 1 and 2. In Part 4, I show you how to repeat the process with an AWS CloudFormation template.

Part 1. The logging function

I created this function to do the heavy lifting of writing the log entries in CloudWatch. Here is the complete function.

function Write-LogsEntry
{
  [CmdletBinding()]
  Param(
    [Parameter(Mandatory=$true)]
    [string] $LogGroupName,
    [Parameter(Mandatory=$true)]
    [string] $LogStreamName,
    [Parameter(Mandatory=$true)]
    [string] $LogString
  )

  #Determine if the LogGroup Exists
  If (-Not (Get-CWLLogGroup -LogGroupNamePrefix $LogGroupName)){
    New-CWLLogGroup -LogGroupName $logGroupName
    #Since the loggroup does not exist, we know the logstream does not exist either
    $CWLSParam = @{
      LogGroupName = $logGroupName
      LogStreamName = $logStreamName
    }
    New-CWLLogStream @CWLSParam
  }
  #Determine if the LogStream Exists
  If (-Not (Get-CWLLogStream -LogGroupName $logGroupName -LogStreamName $LogStreamName)){
    $CWLSParam = @{
      LogGroupName = $logGroupName
      LogStreamName = $logStreamName
    }
    New-CWLLogStream @CWLSParam 
  }
            
  $logEntry = New-Object -TypeName 'Amazon.CloudWatchLogs.Model.InputLogEvent'
  $logEntry.Message = $LogString
  $logEntry.Timestamp = (Get-Date).ToUniversalTime()
  #Get the next sequence token
  $SequenceToken = (Get-CWLLogStream -LogGroupName $LogGroupName -LogStreamNamePrefix $logStreamName).UploadSequenceToken
  
  #There will be no $SequenceToken when a new Stream is created to we adjust the parameters for this          
  if($SequenceToken){
    $CWLEParam = @{
      LogEvent      = $logEntry
      LogGroupName  = $logGroupName
      LogStreamName = $logStreamName
      SequenceToken = $SequenceToken
    }
    Write-CWLLogEvent @CWLEParam 
  }else{
    $CWLEParam = @{
      LogEvent      = $logEntry
      LogGroupName  = $logGroupName
      LogStreamName = $logStreamName
    }
    Write-CWLLogEvent @CWLEParam 
  }
}
  1. We will need to pass in three parameters whenever the function is called.
    • LogGroupName
    • LogStreamName
    • LogString
    Param(
[Parameter(Mandatory=$true)]
[string] $LogGroupName,
[Parameter(Mandatory=$true)]
[string] $LogStreamName,
[Parameter(Mandatory=$true)]
[string] $LogString
)
  2. Next, we determine whether the log group exists. If it does, we reuse it; if not, we create a new one. When a new log group is created, we automatically know we need a log stream within it. 
     #Determine if the LogGroup Exists
  If (-Not (Get-CWLLogGroup -LogGroupNamePrefix $LogGroupName)){
    New-CWLLogGroup -LogGroupName $logGroupName
    #Since the loggroup does not exist, we know the logstream does not exist either
    $CWLSParam = @{
      LogGroupName = $logGroupName
      LogStreamName = $logStreamName
    }
    New-CWLLogStream @CWLSParam
  }
  3. Similar to step 2, we determine whether the log stream exists. This step creates the log stream when a new log group is created, but we need the flexibility to create multiple log streams within the same log group in case we want to separate items within the group. 
      #Determine if the LogStream Exists
  If (-Not (Get-CWLLogStream -LogGroupName $logGroupName -LogStreamName $LogStreamName)){
    $CWLSParam = @{
      LogGroupName = $logGroupName
      LogStreamName = $logStreamName
    }
    New-CWLLogStream @CWLSParam 
  }
  4. Next, we prepare the log entry by defining a new InputLogEvent object, the log Message, and the Timestamp. 
      $logEntry = New-Object -TypeName 'Amazon.CloudWatchLogs.Model.InputLogEvent'
  $logEntry.Message = $LogString
  $logEntry.Timestamp = (Get-Date).ToUniversalTime()
  5. To submit multiple log entries, we need a sequence token. To obtain one, we query CloudWatch. 
      #Get the next sequence token
  $SequenceToken = (Get-CWLLogStream -LogGroupName $LogGroupName -LogStreamNamePrefix $logStreamName).UploadSequenceToken
  6. If this is the first entry, there will be no $SequenceToken, and we need to account for this when submitting our log entries. 
    if($SequenceToken){
    $CWLEParam = @{
      LogEvent      = $logEntry
      LogGroupName  = $logGroupName
      LogStreamName = $logStreamName
      SequenceToken = $SequenceToken
    }
    Write-CWLLogEvent @CWLEParam 
  }else{
    $CWLEParam = @{
      LogEvent      = $logEntry
      LogGroupName  = $logGroupName
      LogStreamName = $logStreamName
    }
    Write-CWLLogEvent @CWLEParam 
  }

Part 2. Using the function

Now that the function is in place, we are ready to begin bootstrapping our instance. For this project, we will perform a couple of simple tasks on the instance, and we will send some data to CloudWatch as the tasks are processed.

  1. First, we prepare the two parameters that will remain static for the duration of our instance initialization.
    • $logGroupName
    • $logStreamName
    $logGroupName = 'MyNewWindowsInstance'
$logStreamName = "*name*-" + (Get-Date (Get-Date).ToUniversalTime() -Format "MM-dd-yyyy" )

    Note: The $logStreamName code “*name*-” looks odd, but it allows you to automate the builds and avoid hardcoded names. I will go into more detail later in this post.

  2. Now we begin to bootstrap our instance. This includes the following tasks.
    1. Create a new user with credentials stored in AWS Secrets Manager
    2. Add the user to the Local Administrators and Remote Management groups
    3. Install a Windows feature and enable PowerShell remoting
    4. Complete the bootstrap

A. Create a new user

The best practice is to never send credentials in the clear, and we make sure to follow this best practice when provisioning the new user. I created a user name and password in AWS Secrets Manager. To retrieve these objects, make the following call to Secrets Manager.

(Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
and
(Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).password

Note:If you’re unfamiliar with Secrets Manager, I encourage you to visit the blog Securing passwords in AWS Quick Starts using AWS Secrets Manager.

New-LocalUser -Name (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username -Password (ConvertTo-SecureString (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).password -AsPlainText -Force)

We log the execution of the task by making a call to our logging function.

Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Created local user'

Another way to write this call is by using “splatting,” which makes the call a bit easier to read.

$splat = @{
  LogGroupName  = $LogGroupName
  LogStreamName = $LogStreamName
  LogStream     = 'Created local user'
}
Write-LogsEntry @splat

B. Add the user to the Local Administrators and Remote Management groups

We will now add the user to the two different groups and log the execution. We will again retrieve the user name from our Secrets Manager store and pass it to our calls.

Add-LocalGroupMember -Group "Administrators" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Administrators Group'
Add-LocalGroupMember -Group "Remote Management Users" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Remote Management Users Group'

C. Install a Windows feature and enable PowerShell remoting

Now let’s take care of the final two tasks. Install the AWS Management Console and enable PowerShell remoting.

Install-WindowsFeature web-mgmt-console
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Installed Windows Feature Web Management Console'
Enable-PSRemoting -Force
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Enabled PowerShell remoting'

D. Complete the bootstrap

We finish with a final CloudWatch entry that indicates the end of UserData processing.

Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'UserData processing complete'

Part 3. Putting it all together with PowerShell

Now that we have all the moving pieces ready, we can put them into a script that will provision an Amazon EC2 instance, execute the UserData tasks, and write the logs to CloudWatch. Before we begin, create an Identity and Access Management (IAM) role that has the following access.

SecretsManagerReadWrite

CloudWatchLogsFullAccess

  1. We begin by adding our code to a $Script variable that will be passed to the EC2 launch command. To pass the code to the variable, we make use of a Here-String.
  2. To send PowerShell code to our EC2 instance’s UserData pipe, we need to specify the language we are sending. This is why the code is bracketed with <powershell/powershell>. 
    $Script = @'
<powershell>

function Write-LogsEntry

    function Write-LogsEntry
    {
      [CmdletBinding()]
      Param(
        [Parameter(Mandatory=$true)]
        [string] $LogGroupName,
        [Parameter(Mandatory=$true)]
        [string] $LogStreamName,
        [Parameter(Mandatory=$true)]
        [string] $LogString
      )
    
      #Determine if the LogGroup Exists
      If (-Not (Get-CWLLogGroup -LogGroupNamePrefix $LogGroupName)){
        New-CWLLogGroup -LogGroupName $logGroupName
        #Since the loggroup does not exist, we know the logstream does not exist either
        $CWLSParam = @{
          LogGroupName = $logGroupName
          LogStreamName = $logStreamName
        }
        New-CWLLogStream @CWLSParam
      }
      #Determine if the LogStream Exists
      If (-Not (Get-CWLLogStream -LogGroupName $logGroupName -LogStreamName $LogStreamName)){
        $CWLSParam = @{
          LogGroupName = $logGroupName
          LogStreamName = $logStreamName
        }
        New-CWLLogStream @CWLSParam 
      }
                
      $logEntry = New-Object -TypeName 'Amazon.CloudWatchLogs.Model.InputLogEvent'
      $logEntry.Message = $LogString
      $logEntry.Timestamp = (Get-Date).ToUniversalTime()
      #Get the next sequence token
      $SequenceToken = (Get-CWLLogStream -LogGroupName $LogGroupName -LogStreamNamePrefix $logStreamName).UploadSequenceToken
      
      #There will be no $SequenceToken when a new Stream is created to we adjust the parameters for this          
      if($SequenceToken){
        $CWLEParam = @{
          LogEvent      = $logEntry
          LogGroupName  = $logGroupName
          LogStreamName = $logStreamName
          SequenceToken = $SequenceToken
        }
        Write-CWLLogEvent @CWLEParam 
      }else{
        $CWLEParam = @{
          LogEvent      = $logEntry
          LogGroupName  = $logGroupName
          LogStreamName = $logStreamName
        }
        Write-CWLLogEvent @CWLEParam 
      }
    }

$logGroupName = 'MyNewWindowsInstance'
$logStreamName = "*name*-" + (Get-Date (Get-Date).ToUniversalTime() -Format "MM-dd-yyyy" )
    
   New-LocalUser -Name (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username -Password (ConvertTo-SecureString (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).password -AsPlainText -Force)

Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Created local user'

Add-LocalGroupMember -Group "Administrators" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Administrators Group'
Add-LocalGroupMember -Group "Remote Management Users" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Remote Management Users Group'

Install-WindowsFeature web-mgmt-console
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Installed Windows Feature Web Management Console'
Enable-PSRemoting -Force
Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Enabled Powersell remoting'

Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'UserData processing complete'

</powershell>
'@
  3. Earlier, we mentioned an odd piece of code called “*name*-“. We want logStreamName to contain the name of the instance. We could simply type the name of the instance here and be done with it. 
    
$logStreamName = "MyWindowsInstance-" + (Get-Date (Get-Date).ToUniversalTime() -Format "MM-dd-yyyy" )

    In fact, that is what I did with the logGroupName.

    $logGroupName = 'MyNewWindowsInstance'

    However, this forces us to edit the code every time we deploy a new instance, and it negates the option to deploy a series of instances on the fly. By adding this construct (a keyword surrounded by asterisks), we can externalize the object (in this case, the “name”) and update it at execution time with a simple replace call.

    $name = "MyWindowsInstance"
$Script = $Script.Replace("*name*", $name)

    The previous code replaces any occurrences of “*name*” with the contents of the $name variable (in this case, “MyWindowsInstance”).

  4. Now that the $Script is ready, we prepare the $UserData variable. 
    $UserData = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($Script))
  5. Using the $name variable defined earlier, we prepare the tagging construct for the instance. 
    #Tags
$tag1 = @{ Key = "Name"; Value = $name }
$tagspec1 = new-object Amazon.EC2.Model.TagSpecification
$tagspec1.ResourceType = "instance"
$tagspec1.Tags.Add($tag1)
  6. We are now ready to combine all the elements needed to call the EC2 execution command. 
    $parms = @{
ImageId             = "ami-057549bd0bba43bc1"
MinCount            = "1"
MaxCount            = "1"
InstanceType        = "t2.large"
UserData            = $UserData
TagSpecification    = $tagspec1
InstanceProfile_Arn = "arn:aws:iam::55555555555:instance-profile/ec2-access"
}
New-EC2Instance @parms

One important note. To retrieve the user ID and password to be used when provisioning the new user, ensure that your instance is associated with a role that has access to Secrets Manager and is able to create CloudWatch entries. The Amazon Resource Name (ARN) for this is added in the InstanceProfile_Arn parameter section.

Part 4. Putting it all together with a CloudFormation template

As we did in the previous section, it’s time to put what we learned together—this time with a CloudFormation template.For this example, I will create the roles required for the instance to talk to CloudWatch and Secrets Manager.

Note that, unlike the previous example, you must specify the name of the CloudWatch logs and the name of the instance ahead of time. You can, however, use parameters to externalize these options.

This template is also available in the quickstart-examples repo.

Resources:
  MyEC2AccessRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - ec2.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
  CWRolePolicies:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: CloudWatchLogsFullAccess
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action: "*"
          Resource: "*"
      Roles:
      - !Ref MyEC2AccessRole
  SMRolePolicies:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: SecretsManagerReadWrite
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action: "*"
          Resource: "*"
      Roles:
      - !Ref MyEC2AccessRole
  MyEC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: "/"
      Roles:
      - !Ref MyEC2AccessRole
  MyEC2Instance: 
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: "ami-057549bd0bba43bc1"
      Tags: 
        - 
          Key: "Name"
          Value: "MyWindowsInstance"
      IamInstanceProfile: 
        !Ref MyEC2InstanceProfile
      UserData: 
        !Base64 |
          <powershell>
            function Write-LogsEntry
                {
                  [CmdletBinding()]
                  Param(
                    [Parameter(Mandatory=$true)]
                    [string] $LogGroupName,
                    [Parameter(Mandatory=$true)]
                    [string] $LogStreamName,
                    [Parameter(Mandatory=$true)]
                    [string] $LogString
                  )
                
                  #Determine if the LogGroup Exists
                  If (-Not (Get-CWLLogGroup -LogGroupNamePrefix $LogGroupName)){
                    New-CWLLogGroup -LogGroupName $logGroupName
                    #Since the loggroup does not exist, we know the logstream does not exist either
                    $CWLSParam = @{
                      LogGroupName = $logGroupName
                      LogStreamName = $logStreamName
                    }
                    New-CWLLogStream @CWLSParam
                  }
                  #Determine if the LogStream Exists
                  If (-Not (Get-CWLLogStream -LogGroupName $logGroupName -LogStreamName $LogStreamName)){
                    $CWLSParam = @{
                      LogGroupName = $logGroupName
                      LogStreamName = $logStreamName
                    }
                    New-CWLLogStream @CWLSParam 
                  }
                            
                  $logEntry = New-Object -TypeName 'Amazon.CloudWatchLogs.Model.InputLogEvent'
                  $logEntry.Message = $LogString
                  $logEntry.Timestamp = (Get-Date).ToUniversalTime()
                  #Get the next sequence token
                  $SequenceToken = (Get-CWLLogStream -LogGroupName $LogGroupName -LogStreamNamePrefix $logStreamName).UploadSequenceToken
                  
                  #There will be no $SequenceToken when a new Stream is created to we adjust the parameters for this          
                  if($SequenceToken){
                    $CWLEParam = @{
                      LogEvent      = $logEntry
                      LogGroupName  = $logGroupName
                      LogStreamName = $logStreamName
                      SequenceToken = $SequenceToken
                    }
                    Write-CWLLogEvent @CWLEParam 
                  }else{
                    $CWLEParam = @{
                      LogEvent      = $logEntry
                      LogGroupName  = $logGroupName
                      LogStreamName = $logStreamName
                    }
                    Write-CWLLogEvent @CWLEParam 
                  }
                }

            $logGroupName = 'MyNewWindowsInstance'
            $logStreamName = "MyWindowsInstance" + (Get-Date (Get-Date).ToUniversalTime() -Format "MM-dd-yyyy" )
                
              New-LocalUser -Name (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username -Password (ConvertTo-SecureString (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).password -AsPlainText -Force)

            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Created local user'

            Add-LocalGroupMember -Group "Administrators" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Administrators Group'
            Add-LocalGroupMember -Group "Remote Management Users" -Member (ConvertFrom-Json -InputObject (Get-SECSecretValue -SecretId LocalBuildCredenitals).SecretString).username
            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Added local user to Remote Management Users Group'

            Install-WindowsFeature web-mgmt-console
            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Installed Windows Feature Web Management Console'
            Enable-PSRemoting -Force
            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'Enabled PowerShell remoting'

            Write-LogsEntry -LogGroupName $LogGroupName -LogStreamName $LogStreamName -LogString 'UserData processing complete'
                      </powershell>

The results

While the instance is starting up and executing the UserData section, you can head over to CloudWatch and observe the logs being generated.

The following image shows all the entries created.

Conclusion

In this blog post, I covered a technique to help you externalize what is happening with your Amazon EC2 instance at creation time. I showed you how to leverage AWS Secrets Manager to ensure credentials are secure and how you can make the code reusable by externalizing the elements you want to persist on a per-instance case with PowerShell. Finally, I created an AWS CloudFormation template that makes use of our UserData scripts. While this example is geared specifically toward Windows Amazon EC2 instances, you can, with a few changes, adapt this code to work with Linux builds.

I hope you find this post helpful. If you have any questions, let us know in the comments. Happy automating!