Get Disk Utilization of Your Fleet Using EC2 Systems Manager Custom Inventory Types

Note: There is a new version to this solution. Please check our new blog.

Amazon EC2 Systems Manager Inventory provides a centralized way to collect and query system, application, and instance metadata. Using the resource data sync feature, you can sync this metadata to Amazon S3. In Amazon S3 you can aggregate the metadata for different AWS Regions and accounts. After you sync this inventory data to Amazon S3, you can create various visuals of the data using Amazon Athena and Amazon QuickSight.

The inventory data collection policy is configured using State Manager , which in turn gets executed by aws:softwareInventory plugin in amazon-ssm-agent.

Amazon EC2 Systems Manager Inventory provides two ways to define the types of data that it collects: predefined and custom.

· Predefined data types (with prefix AWS) are natively supported by the inventory plugin via multiple gatherers. Some examples of predefined inventory types are AWS:Application and AWS:WindowsUpdate.

· Custom data type (with prefix Custom) is a special inventory data type that can be defined by end users. This data type provides the flexibility of collecting additional inventory data, such as server rack location of a managed instance.

In this blog, I’ll walk you through an example that shows how to use the custom inventory data type to collect disk utilization for Windows instances. We’ll use PowerShell scripts to collect disk utilization data in the Inventory. After the data is collected, we’ll use this data to get fleet-level aggregation of disk usage.

Step1: Create Inventory Policy Document with aws:runPowerShellScript plugin

By default, you can use AWS-GatherSoftwareInventory document to collect Inventory data. However, for this example we’ll create a document with aws:runPowerShellScript & aws:softwareInventory plugins to ensure that a PowerShell script is invoked before Inventory collection begins. Since amazon-ssm-agent preserves the order of the plugin’s executions, it ensures that the latest Disk Utilization data is captured before Inventory data is collected.

Option 1: Create Document using the AWS Management Console

You can create the document in the AWS console by going to EC2 -> Systems Manager Shared Resources -> Documents. Here is the document content.

{
    "schemaVersion": "2.2",
    "description": "Run first a shell script & then inventory plugin.",
    "mainSteps": [
        {
            "action": "aws:runPowerShellScript",
            "name": "runPowerShellScript",
            "inputs": {
                "runCommand": "{{ commands }}"
            }
        },
        {
            "action": "aws:softwareInventory",
            "name": "collectSoftwareInventoryItems",
            "inputs": {
                "applications": "{{ applications }}",
                "awsComponents": "{{ awsComponents }}",
                "networkConfig": "{{ networkConfig }}",
                "windowsUpdates": "{{ windowsUpdates }}",
                "customInventory": "{{ customInventory }}"
            }
        }
    ],
    "parameters": {
        "commands": {
            "type": "StringList",
            "description": "(Required) Specify a shell script or a command to run.",
            "minItems": 1,
            "displayType": "textarea"
        },
        "applications": {
            "type": "String",
            "default": "Enabled",
            "description": "(Optional) Collect data for installed applications.",
            "allowedValues": [
                "Enabled",
                "Disabled"
            ]
        },
        "awsComponents": {
            "type": "String",
            "default": "Enabled",
            "description": "(Optional) Collect data for AWSComponents like amazon-ssm-agent.",
            "allowedValues": [
                "Enabled",
                "Disabled"
            ]
        },
        "networkConfig": {
            "type": "String",
            "default": "Enabled",
            "description": "(Optional) Collect data for Network configurations.",
            "allowedValues": [
                "Enabled",
                "Disabled"
            ]
        },
        "windowsUpdates": {
            "type": "String",
            "default": "Enabled",
            "description": "(Optional) Collect data for all WindowsUpdates.",
            "allowedValues": [
                "Enabled",
                "Disabled"
            ]
        },
        "customInventory": {
            "type": "String",
            "default": "Enabled",
            "description": "(Optional) Collect data for custom inventory.",
            "allowedValues": [
                "Enabled",
                "Disabled"
            ]
        }
    }
}

Option 2: Create Document using the AWS command line (aws-cli)

a. Create the document by running the aws-cli command that creates the document:

aws ssm create-document --content file://path to your file\FileName --name "CustomInventory-Doc" --document-type Command

b. Verify the document’s status by running following command:

aws ssm list-documents --document-filter-list key=Name,value= CustomInventory-Doc

Here is a sample output that you should see:
{
    “DocumentIdentifiers”: [
        {
            “Name”: ” CustomInventory-Doc “,
            “PlatformTypes”: [
                “Windows”,
                “Linux”
            ],
            “DocumentVersion”: “1”,
            “DocumentType”: “Command”,
            “Owner”: “xxx”,
            “SchemaVersion”: “2.0”
        }
    ]
}

Step2: Create association using CustomInventory-Doc

Now that the inventory policy document is created, we will “Create Association,” that is we’ll associate this policy document to the targeted instances. To use the EC2 console to Create Association, go to State Manager under Systems Manager Service.

Create Association

Next, you need to pick instances to which you want to attach this Association. In addition, define a schedule for this inventory collection. After you pick the instances and schedule, you can paste following PowerShell script in the Commands parameter (as shown in the screenshot that follows). This script is executed by the aws:runPowerShellScript plugin before the Inventory plugin is invoked.

Script:

$data = get-wmiobject win32_logicaldisk | Select-Object @{n="DeviceId";e={$_."DeviceID"}}, @{n="VolumeName";e={$_."VolumeName"}}, @{n="Use%";e={"{0}" -f [math]::Round(($_."Size" - $_."FreeSpace") * 100 / $_."Size",0)}}, @{n="Size(GB)";e={"{0}" -f [math]::Round($_."Size" / 1GB ,0)}} | ConvertTo-Json
$content = "{`"SchemaVersion`" : `"1.0`", `"TypeName`": `"Custom:DiskUtilization`", `"Content`": $data}"
$instanceId = Invoke-RestMethod -uri http://169.254.169.254/latest/meta-data/instance-id
$filepath = "C:\ProgramData\Amazon\SSM\InstanceData\" + $instanceId + "\inventory\custom\CustomDiskUsage.json"
if (-NOT (Test-Path $filepath)) {
    New-Item $filepath -ItemType file
}
Set-Content -Path $filepath -Value $content

This script gets disk utilization data using win32_logicaldisk. It uses Instance-Metadata to get InstanceId, which is required in order to save the content in the path: %SystemDrive%\ProgramData\Amazon\SSM\InstanceData\<instance-id>\inventory\custom

Choose Create Association and that’s it. After the policy runs on the targeted instances, disk utilization data will be collected and become ready for consumption.

Let’s go to Managed Instances and check disk utilization for an instance by choosing the Inventory tab. The following screenshot shows the new Custom data for one of my instances.

We can also apply various filters to determine the fleet’s Disk Utilization health.

Let’s apply a filter to list all instances with disk utilization of more than 50 percent. The following screenshot shows instances matching the filter.

Conclusion

This blog shows you how to create a custom document with aws:runPowerShellScript & aws:softwareInventory plugins. This allows you to collect and then send Custom Inventory data from an instance every time the Inventory policy is run. The data can then be queried at both the fleet and instance level.

In this blog, we used a PowerShell script to get the custom data, however the same document can also be used to trigger any third-party application running in the instance to collect custom data.

About the Author

Saurabh Shankar is a Software Development Engineer with the Amazon EC2 Systems Manager team. He has been with Amazon for four years, working on Inventory and other features of Amazon EC2 Systems Manager. Outside work, he enjoys trekking and taking photographs.

AWS Cloud Operations Blog