Use AWS Systems Manager for VMware Cloud on AWS (VMC) operations management

A hybrid cloud strategy creates management and governance challenges for our customers. These challenges include maintaining consistent cloud security and compliance policies across hybrid VMC and cloud environments, providing a single pane of glass for visualizing and acting on operational data, and providing deployment automation and control of cloud infrastructure across multiple cloud environments.

VMware Cloud on AWS (VMC) lets you quickly migrate VMware workloads to a VMware-managed Software-Defined Data Center (SDDC) running in AWS and extend your on-premises data centers without replatforming or refactoring applications. It enables customers to deploy VMware’s SDDCs and consume vSphere workloads on AWS’s global infrastructure as a managed service. You can use native AWS services with Virtual Machines (VMs) in the SDDC, to reduce operational overhead and lower your Total Cost of Ownership (TCO) while increasing your workload’s agility and scalability.

AWS Systems Manager (SSM) is a secure end-to-end management solution for AWS, hybrid, or multi-cloud environments. In this solution, we’ll demonstrate how Systems Manager’s node management capabilities can be used to remotely manage your compute in a hybrid VMC environment.

In this solution, we’ll demonstrate how Systems Manager provides centralized node management, including collecting inventory and initiating secure sessions. We’ll demonstrate how patch automation for Amazon EC2 works seamlessly for VMC as well as demonstrate automation for both installation and subsequent provisioning of 3rd party packages to VMC VMs from a centralized interface.

Installation

VMC setup

There are two required accounts when setting up VMC. First is the VMware Cloud SDDC account. This is an AWS account that runs the SDDC or VMC resources. It’s owned and operated by VMware. The second required account is an AWS customer-owned account. To successfully attach this account to the SDDC, it should have at least one Amazon Virtual Private Cloud (Amazon VPC) within that account. We call this the Connected VPC. During the provisioning of the VMware Cloud SDDC account by following the steps outlined in deploying VMC SDDC, AWS Elastic Network Interfaces (ENI) are set up in this connected VPC to provide high bandwidth and low latency access to AWS services within this VPC.

In our solution, we start with a setup of a VMware Cloud SDDC account and a Connected VPC account. In our workload account, we setup a VPC and connect it via an AWS Transit Gateway attachment to an AWS Transit Gateway shared with our VPC using AWS Resource Access Manager (AWS RAM) from the Connected VPC. This Transit Gateway is then peered with VMware Transit Connect – a white-labeled Transit Gateway provided by VMware, and this peering connects our VPC in the workload account with the VMware Cloud SDDC.

We deploy the Systems Manager agents on the VMC VMs in the VMWare Cloud SDDC and connect it to the Systems Manager service running in the customer’s workload account.  With this setup, Systems Manager’s node management capabilities can be used to remotely manage your compute in a hybrid VMC environment for both centralized node management as well as operations management capabilities

Systems Manager setup

Setup Systems Manager for hybrid environments – Follow these steps to configure Systems Manager to provide centralized operations and management for a hybrid environment, including VMs in other cloud environments. After you finish configuring your VMC VM for Systems Manager, the IDs of your hybrid managed node (i.e., the VMC VM) will be distinguished from Amazon Elastic Compute Cloud (Amazon EC2) instances with the prefix “mi-“. Amazon EC2 instance IDs use the prefix “i-“.

The following diagram shows the solution architecture diagram for our setup.

Figure1: Architecture diagram describing setup of centralized operations management using Systems Manager for VMware Cloud on AWS

Fleet Management

Fleet Manager, a capability of Systems Manager, lets you drill down to individual nodes (services, devices, or other resources) to perform common system management tasks, such as disk and file exploration, log management, and user management from a console.

Navigate to the Systems Manager console, and select Fleet Manager on the left panel. On the Managed nodes panel in the main console, select the VMC VM with the prefix “mi-“. When you drill down on the managed node, you can view information about the folder and file data stored on the volumes attached to your VM. This includes information such as performance data about your instances in real-time, as well as managing operating system (OS) user accounts on your VM.

Figure 2: VMC VM as a managed node using Systems Manager Fleet Manager

Figure 3: VMC VM node details using Systems Manager

Patch Management

Patch Manager, a capability of Systems Manager, automates the process of patching managed nodes with both security related and other types of updates.

Navigate to the Systems Manager console, and select Fleet Manager on the left panel. On the Managed nodes panel in the main console, select the VMC VM with the prefix “mi-“ and perform the following steps

• Step 1: Select Tags from the bottom panel and add a tag key and value to the VMC VM.
• Step 2: Follow instructions here to add managed nodes to a patch group using tags and ensure to use the same tag key from Step 1 as the tag value for the PatchGroup tag key

Navigate to the AWS CloudFormation console and launch the aws-patch-manager-v1.yaml CloudFormation template. For Parameters, provide the tag key from Step 1 above as the value for the VMCTagKey parameter

The template creates a Systems Manager State Manager association that runs weekly and uses the ‘AWS-RunPatchBaseline’ automation document to patch the tagged VMC VM with patch baselines associated with its patch group.  Once the template has deployed successfully, you can test patching of your VMC VM by navigating to the State Manager console and then on the right panel, selecting the Association with a name that has the ‘PatchVMsWeekly’ suffix.  Select the  Apply Association now button to start on demand patching of your VMC VM.

The following diagrams show patching status of our VMC VM progressing from a non-compliant to successful status:

Figure 4: VMC VM showing a non-compliant patching status with Systems Manager Patch Manager prior to the patch baseline being applied

Figure 5: VMC VM showing patching in progress with Systems Manager Patch Manager

Figure 6: VMC VM showing successful patching with Systems Manager Patch Manager

Figure7: VMC VM showing successful patching with Systems Manager Patch Manager compliance reporting

Package distribution

Customers routinely leverage 3rd party agent-based packages and vulnerability management tools, such as CrowdStrike, TrendMicro, and Tenable to secure their AWS environments. AWS supports distribution of third-party agents with AWS Systems Manager Distributor (Distributor). Distributor lets you package your own software or finds AWS-provided agent software packages, such as AmazonCloudWatchAgent, to install on AWS Systems Manager managed instances including your VMC environments.

Follow these instructions to create a package for your 3rd party software and upload it to Amazon S3. In this blog post we use an Amazon S3 bucket named s3-examplepackage-[accountid]-[region]. We demonstrate the solution using the Example Package from the step wise instructions that has been uploaded to our S3 bucket in the example package folder. The example package includes a completed JSON manifest and three .zip files. The following diagram illustrates our custom package uploaded to S3:

Figure 8: S3 bucket with custom package to be installed on VMC VM

Navigate to the AWS CloudFormation console and follow the instructions here to launch a CloudFormation stack using the aws-centralizedssmdistributor-v1 CloudFormation template. This template makes your custom package available as a Systems Manager Automation document under the Owned by me tab in the Distributor console in each member account in your AWS Organization. It then provisions a State Manager association in each member account that installs the package in that account as per the schedule and tags specified in the association. The template takes the following parameters.

• PackageName: Name for your package
• S3PackageBucket: Name of the S3 Bucket where the package contents are uploaded (for e.g.  s3-examplepackage-[accountid]-[region])
• S3PackageBucketFolder: Name of the S3 Bucket Folder where the manifest is uploaded (for e.g. SimplePackage2)
• S3PackageUrl: Https URL of the bucket including prefix where the package contents are uploaded (for e.g. https://s3-examplepackage-[accountid]-[region].s3 [region]. amazonaws.com/examplepackage)
• Version: Provide the exact version name from the manifest file (for e.g. 1.0.2)
• AssociationName: Name for your association
• Action: Specify whether to install or uninstall the package. (for e.g. Install or Uninstall)
• InstallationType: Specify the type of installation (for e.g. In-place update)
• OutputS3Prefix: The S3 Key Prefix used for AWS Systems Manager Run Command Output (Default ‘’)
• ScheduleExpression: The Schedule Expression for the AWS Systems Manager Association. (for e.g. “rate(30 minutes)”)
• TargetResourceTagKey: The AWS Systems Manager Tag Key for the target (provide the VMC VM tag key if you like the package to be installed only on the VMC VM)
• TargetResourceTagValue: The AWS Systems Manager Tag Value for the target (provide the value of the VMC VM tag key if you like the package to be installed only on the VMC VM)

Once the template has deployed successfully, navigate to the AWS Systems Manager console and select Distributor from the left panel. Select the Owned by me tab and validate that your examplepackage is available there.  Select the package (SimplePackage2) and validate the Version from the Details section. Validate that the Attachments information in the Additional Information section contains the zip files and hash exactly as in the manifest file for your custom package (examplepackage). Here’s our custom package (SimplePackage2) that is now available in Distributor:

Figure 9: Figure illustrating how our custom package is made available in Distributor under the Owned by me tab in each member account

Session Management

Session Manager, a capability of Systems Manager, provides secure and auditable node management without needing to open inbound ports, maintain bastion hosts, or manage SSH keys.  Administrators can grant and revoke access to your VMC VM from a single location, as well as provide one solution to users for Linux, macOS, and Windows Server managed nodes in a multi-cloud environment. Users can connect to the managed node (e.g. VMC VM) across clouds with just one click from the browser or AWS Command Line Interface (AWS CLI) without having to provide SSH keys.

Figure 10: Figure illustrating secure session access with Windows VMC VM

Inventory

Inventory, a capability of Systems Manager, collects metadata from your managed nodes running on AWS – either on-premises or on other clouds. The metadata includes applications (application names, publishers, versions), files (name, size, version, installed date, modification, last accessed times), network configurations (IP address, MAC address, DNS, gateway, subnet mask), etc. Access the full list of metadata types collected by Systems Manager Inventory here.

1. Navigate to the Systems Manager console, and in the navigation pane, select Inventory. The data in the Systems Manager console on the Inventory page includes several predefined cards to help you query the data.

Figure 11: AWS Systems Manager Inventory displays predefined cards to help you query inventory metadata on your VMC VM.

Cleanup

To avoid recurring charges, and to clean up your account after trying the solution outlined in this post, perform the following:

1. Follow these steps to uninstall the Systems Manager agent on your VMC VM.
2. Delete the cloudformation stack for the aws-centralizedssmdistributor-v1 template
3. Delete the s3-examplepackage-[accountid]-[region] Amazon S3 bucket that was created for this solution.

Conclusion

Cloud Operations services can provide a unified operational view and an optimized IT infrastructure to alleviate your management, orchestration, and portability challenges across clouds. Systems Manager, a Cloud Operations service, provides node management capabilities that can be used to remotely manage your compute in a hybrid environment. In this post, we demonstrated how you can use Systems Manager to collect inventory, initiate secure sessions as well as centralize and automate patch management and package distribution for your compute running in VMware Cloud on AWS (VMC).

About the authors