AWS Machine Learning Blog

Launch EI accelerators in minutes with the Amazon Elastic Inference setup tool for EC2

Note: Amazon Elastic Inference is no longer available. Please see Amazon SageMaker for similar capabilities.

The Amazon Elastic Inference (EI) setup tool is a Python script that enables you to quickly get started with EI.

Elastic Inference allows you to attach low-cost GPU-powered acceleration to Amazon EC2 and Amazon SageMaker instances to reduce the cost of running deep learning inference by up to 75 percent. If you are using EI for the first time, there are a number of dependencies that must be set up: Amazon Web Services (AWS) PrivateLink VPC Endpoints, IAM policies, and security group rules. To accelerate this, The EI setup script makes it easy for you to get started by creating the necessary resources to help you launch EI accelerators in minutes. In this blog post I describe how to use the script, functionalities of the script, and what to expect when you run it.

At a high level, the script does the following:

  1. Creates an IAM role for the instance with an IAM policy that lets you connect to the AWS Elastic Inference service.
  2. Creates a security group with the necessary ingress and egress rules to allow the instance to communicate with the accelerator.
  3. Creates an AWS PrivateLink VPC Endpoint within your desired subnet.
  4. Launches the desired EC2 instance with an EI accelerator using the latest AWS Deep Learning AMI (DLAMI) for the chosen operating system

Prerequisites

To set up EI, run the script linked below. It depends on the following entities:

  1. Python 3 installed on your local machine where you expect to run the tool.
  2. The AWS SDK for Python (Boto3).
  3. An Amazon VPC in the Region where you are launching the instance (could be your default VPC).
  4. Subnet where you’d like to launch the instance.
  5. EC2 Key Pair.
  6. AWS credentials.

With these in place, download the amazonei_setup.py script from GitHub to your local machine and run it from your terminal using following command:

$ python amazonei_setup.py

What the tool creates on your behalf

The script creates following AWS resources:

  • Instance role with an Amazon EI Policy. This role is created the first time the script is run. In all subsequent runs, script reuses this IAM role. If this role is deleted, script recreates the role next time it is run. The IAM role has following properties:
    • Role name: Amazon-Elastic-Inference-Connect-Role
    • Policy name: Amazon-Elastic-Inference-Connect-Policy
    • Instance profile name: Amazon-Elastic-Inference-Instance-Profile

    The policy description is as follows:

    { "Version": "2012-10-17", 
  "Statement": [
       {
            "Effect": "Allow", 
            "Action": [ 
            "elastic-inference:Connect", 
            "iam:List*",
            "iam:Get*",
            "ec2:Describe*",
            "ec2:Get*" 
            ],
            "Resource": "*"
        } 
    ] 
}
  • Security Group (SG). The security group associated with the EC2 instance should allow inbound traffic to port 443 as required by Amazon EI service. You also need inbound rules that allow traffic to port 22 for SSH. If a security group matching these rules is found, it is used. However, if no matching SG is found, a new SG with required rules is created. The outbound rules are set to allow traffic to all ports. The new SG name is amazon_ei_security_group, with the description Security Group for accessing Amazon EI service.
  • Interface VPC endpoint (AWS PrivateLink). The script scans for existing endpoint associated with Amazon EI service for the Region and VPC that you chose. For example, for the us-west-2 Region, the script looks for the endpoint with name amazonaws.us-west-2.elastic-inference.runtime in the given VPC ID.  If the endpoint is not found, the script creates one. Also, the script sets following attributes of the VPC endpoint to true, as required by Amazon EI:
    • EnableDnsSupport
    • EnableDnsHostNames
    • The script modifies the endpoint and add SG and chosen subnet if they are missing from the discovered endpoint
  • The script discovers latest AWS DLAMI based on the operating system chosen by the user.
  • If all steps succeed, the script launches an instance and reports the instance ID.
  • The script tries to obtain public DNS name after the instance is launched and is in running state.
  • Even if the instance is running, it may not be ready for accepting SSH connection and users may want to wait until the instance is fully initialized. EC2 console or AWS CLI can be used to query the initialization state, using the instance ID that is reported by the script for the newly launched instance.

What to expect when you run the tool

The example here illustrates what to expect when you run the script.

  • Launch the script. The script can be launched from the command prompt as:
  • $ python amazonei_setup.py –region us-west-2 –instance-type m5.xlargeAWS credentials are required to create or modify AWS resources. It uses Boto3, AWS SDK for Python. In order to be able to configure and manage AWS resources, the script needs user credentials. If the script is run without appropriate credentials, it reports the error below: 
    $ python amazonei_setup.py --region us-west-2 --instance-type m5.xlarge
Error setting up Amazon EI configuration - 
 Failed to retrieve VPC endpoints for us-west-2 : An error occurred (RequestExpired)
 when calling the DescribeVpcEndpointServices operation: Request has expired.

    The solution is to configure AWS credentials using one of the methods described in the Amazon Boto3 documentation. After the credentials are in place, the script is able to proceed.

  • Choose Operating System. The script prints informative message and prompts for choosing the OS. It also informs that entering ‘q’ causes the script to exit. Choose ‘1’ for the next step. 
    $ python amazonei_setup.py --region us-west-2 --instance-type m5.xlarge

This script launches Amazon EC2 instances with Amazon Elastic Inference accelerators.
Performs the following functions:
 1. It uses the Deep Learning AMIs preconfigured with EI-enabled deep learning 
 frameworks to launch the instances.
 2. It creates security groups for the instance and VPC endpoint.
 3. It creates the VPC endpoint needed for your instances to communicate with EI 
 accelerators.
 4. It creates an IAM Instance Role and Policy with the permissions needed to 
 connect to accelerators.

 To begin, please choose the Operating System for your instance by typing its index :

 0: Amazon Linux
 1: Ubuntu

Type 'q' to quit.
amazonei-wizard>
  • Choose Accelerator size. The script discovered latest DL AMI for Ubuntu, it also discovered one key pair. If it discovers multiple key pairs, it lists those and ask the user to choose desired key pair by typing its index.  In general if there are multiple eligible inputs, the script shows them as indexed list and let the user choose an item by typing its index. Thus, script lists supported accelerator sizes and lets user choose. 
    amazonei-wizard>1 
Using Image ID: ami-0027dfad6168539c7,Image Name: Deep Learning AMI (Ubuntu 16.04) Version 27.0
Using instance type: m5.xlarge
Using Key Pair: Efti-Default-KeyPair

Please type index of the accelerator type to use: (Note: Please select eia2. This is our latest family of accelerators with double the memory and lower costs than eia1. Note that you will have to use eia1 in ap-northeast-1 Region)

0: eia2.medium (2 GB of accelerator memory)
1: eia2.large (4 GB of accelerator memory)
2: eia2.xlarge (8 GB of accelerator memory)
3: eia1.medium (1 GB of accelerator memory)
4: eia1.large (2 GB of accelerator memory)
5: eia1.xlarge (4 GB of accelerator memory)

Type 'q' to quit.
amazonei-wizard>
  • Choose VPC. As illustrated, user chose option ‘1’ for Accelerator size and the script confirmed the Accelerator size chosen and proceeded to discover IAM role. Subsequently, it presents list of available VPCs. 
    amazonei-wizard> 1 
Using Amazon EI accelerator type: eia2.large

Found an IAM role configured for connecting to Amazon EI service. Name - Amazon-Elastic-Inference-Connect-Role, ARN - arn:aws:iam::326228132093:role/Amazon-Elastic-Inference-Connect-Role

Please select the VPC to use by typing the desired VPC index. Type 0 for default VPC.

0: VPC Id 'vpc-d7d218af'
1: VPC Id 'vpc-0c2496c51925ff1be'

Type 'q' to quit.
amazonei-wizard>
  • Launch an instance. Once user chooses the VPC ID, the script found a security group with matching inbound rules associated with chosen VPC, it also found one subnet associated with the chosen VPC ID. Additionally it found VPC endpoint for Amazon EI service. As the script has all the details to launch an EC2 instance, the script summarizes all the parameters it uses to launch the instance. 
    amazonei-wizard>1 
Using VPC ID: vpc-0c2496c51925ff1be
Using Security Group: sg-00aec97685affb306
Using Subnet: subnet-04881d24764d6e73f

Discovered VPC endpoint for Amazon EI service, ID: vpce-0d2942a8147305240

The script will now launch new instance with following configuration. Type 'y' to continue. 

Accelerator Type: eia2.large
Region: us-west-2
Image-ID: ami-0027dfad6168539c7 - (Deep Learning AMI (Ubuntu 16.04) Version 27.0)
Instance Type: m5.xlarge
Key Pair: Efti-Default-KeyPair
Security Group ID: sg-00aec97685affb306
Subnet ID: subnet-04881d24764d6e73f
Instance Profile: Amazon-Elastic-Inference-Instance-Profile

Type 'y' to continue. Type 'q' to quit.
amazonei-wizard>
  • Launch and wait for the instance to reach running state. As the user typed ‘y’, the script proceeded to launch the instance. The script also printed probable SSH command. The script infers the SSH command based on the OS type, key pair chosen, and the public DNS name. Actual command differs based on location of pem file. The script also warns that the instance may not be immediately accessible via SSH, even though it is in running state. The instance needs to be initialized fully, specifically  the SSH daemon needs to be started before it can accept SSH connections. If the pem file is correctly located the user should be able to access the instance and proceed with using Amazon Elastic Inference. 
    amazonei-wizard>y

 Launching Instance ..

 Launched instance successfully. The instance ID is 'i-0969820364c038cca'.

 Waiting for instance to reach running state ...

 You can use the following sample SSH command to connect to your instance: ssh -i "Efti-Default-KeyPair.pem" ubuntu@ec2-52-13-194-188.us-west-2.compute.amazonaws.com


 Note: Please wait until instance is fully initialized and ready to accept SSH connections. You may check instance status at EC2 console.
 Also please locate your private key file 'Efti-Default-KeyPair.pem'.

amazon-elastic-inference-tools $

Summary

The setup script simplifies your launch of an EC2 instance with EI. It ensures that all settings are correctly configured and instance is launched with requisite permissions to use EI. If you have any feedback about this blog post, feel free to use the comment section on this page.

About the Authors

Eftiquar Shaikh is Senior Software engineer with AWS AI. He works on building AWS services in AI space. When he is not programming, he likes to read, run and travel.

 

 

 

 

Satadal Bhattacharjee is Principal Product Manager with AWS AI. He leads the Machine Learning Engine PM team working on projects such as SageMaker Neo, AWS Deep Learning AMIs, and AWS Elastic Inference. For fun outside work, Satadal loves to hike, coach robotics teams, and spend time with his family and friends.