Deploy Gluon models to AWS DeepLens using a simple Python API

Today we are excited to announce that you can deploy your custom models trained using Gluon to your AWS DeepLens. Gluon is an open source deep learning interface which allows developers of all skill levels to prototype, build, train, and deploy sophisticated machine learning models for the cloud, devices at the edge, and mobile apps.

With Gluon, you can build machine learning models using a simple Python API and a range of pre-built, optimized neural network components. This makes it easy to build neural networks using simple code without sacrificing training performance. Gluon makes building new computer vision models easy; just create your model in SageMaker, and with a single click deploy it to your DeepLens, where the model optimizer will automatically optimize it for the best performance on the device.

In this post, we will walk you through developing a deep neural network model in Amazon SageMaker to detect the direction of the head and deploy it to AWS DeepLens. When there is a person in front of us, we humans can immediately recognize the direction in which the person is looking. For example, the person might be facing straight toward you, or the person might be looking somewhere else. The direction is defined as the head pose. We are going to develop a convolutional neural network mode (CNN) to estimate the head pose using images of human heads. The different head poses are classified as follows: down right, right, up right, down, middle, up, down left, left, and up left. Detecting the head pose could be used to understand who is paying attention in a classroom setting, viewer behavior in advertising, and even in driver assistance systems.

Gluon, the imperative interface in Apache MXNet, offers four major advantages over the symbolic MXNet. First, Gluon offers a full set of plug-and-play neural network building blocks such as predefined layers, optimizers, and initializers. Second, it also allows us to bring the training algorithm and model closer together, which provides flexibility in the development process. Third, it enables developers to define their dynamic neural network models so that they can be built on the fly using Python’s native control flow. Finally, Gluon provides all the benefits without sacrificing the training speed that the underlying engine provides.

Data: Prima Project head-pose image database

First, let’s identify the head-pose dataset we are going to use for the project. For this blog post we will use the Prima Project head pose images. You will find the original raw data in the following link:

Dataset: http://www-prima.inrialpes.fr/perso/Gourier/Faces/HPDatabase.html

Citation:

N. Gourier, D. Hall, J. L. Crowley
Estimating Face Orientation from Robust Detection of Salient Facial Features
Proceedings of Pointing 2004, ICPR, International Workshop on Visual Observation of Deictic Gestures, Cambridge, UK

There are a total of 2,790 head pose images and their corresponding tilt and pan angle attributes in the dataset. The tilt is defined as the north-south vertical axis, and the pan is defined as the east-west horizontal axis.

The dataset is composed of head-pose data for fifteen different individuals. Thus, there are 186 images for each subject (2,790/15 = 186). In this dataset, the head pose is categorized into 9 and 13 discrete tilt and pan angles, respectively (Tilt angles: -90°, -60°, -30°, -15°, 0°, +15°, +30°, +60°, and +90° from head-down posture to head-up posture. Pan angles: -90°, -75°, -60°, -45°, -30°, -15°, 0°, +15°, +30°, +45°, +60°, +75°, and +90° from the observer’s right to the left). When a subject is looking straight into a camera, both tilt and pan angles are 0°. The original image dimensions are 384 x 288 pixels.

Preprocessing the image data

Next, we preprocess the image data that will be used to train a neural network. We have a Python script (python2 preprocessingDataset_py2.py) for the preprocessing.

Run the following command to prepare the input data to generate HeadPoseData_trn_test_x15_py2.pkl (6.7 GB).

python2 preprocessingDataset_py2.py --num-data-aug 15 --aspect-ratio 1

This command lets you generate input images with the dimensions of 84 x 84 pixels and their corresponding head-pose angles.

During the preprocessing, applying the same scaling factor in both height and width of an input image is crucial for the head-pose estimator. If the scaling factors in two axes are different, the head-pose angle is altered. We are mainly targeting two different aspect ratios (1:1 and 16:9). (Spoiler alert! The aspect ratio of full frame size in an AWS DeepLens device is 16:9. Thus, if you want to use the entire frame data from AWS DeepLens for inference, the aspect ratio of 16:9 will be your choice. We used the model trained with the aspect ratio of 1:1 for our final product. Our final product only takes a part of frame data in a square shape for inference.) Inside the preprocessing script, the original head images were cropped and resized into a target image size while applying the same scaling factor in two orthogonal axes (84 x 84 pixels and 96 x 54 pixels for aspect ratios of 1:1 and 16:9, respectively).

The following figure demonstrates the image preprocessing procedure in which the target aspect ratio is 16:9. First, a rectangular crop region with arbitrary length was applied to each image based on the following three criteria: (1) its aspect ratio is 16:9, (2) it must fully contain a face region inside, and (3) it must be contained within the image frame. The selected area is then resized into 96 x 54 pixels. This preprocessing procedure mimics the digital (as well as optical) zoom in a camera.

This preprocessing was repeated 15 times for the data augmentation.

Head-pose classification

Next, we prepare the label data from head-pose angles. The head pose was classified into nine categories (the combinations of three tilt and three pan classes). The head pose contained within ± 19.5° in tilt and pan angles is labeled as a center position (Head pose Class of 4, Tilt Class of 1, and Pan Class of 1). The rationale behind the selection of threshold angles is that sin(19.5°) is equal to 0.33. Therefore, these two angles split a semicircle (the distance between sin(-90°) and sin(90°)) into three equal arc lengths.

Train the ResNet-50 model using Gluon

Creating the nine labels reduced the head-pose problem into a simple image classification task (that is, using an image as input, estimating one head pose out of nine). The model is fine-tuned from a ResNet-50 that we obtained from the MXNet model zoo. There are five main parts in the sample notebook: (1) data loading, (2) additional data augmentation, (3) fine-tuning ResNet-50, (4) validation, and (5) inference. Parts (1), (2), and (3) are especially important for training the model.

Head-Pose Gluon Tutorial Notebook:
https://github.com/aws-samples/headpose-estimator-apache-mxnet/blob/master/HeadPose_ResNet50_Tutorial_Gluon.ipynb

ResNet-50 Model from model zoo

We first download a pre-trained ResNet-50 model. Here is how you load the pre-trained model on Gluon.

Obtain a pre-trained ResNet Model from model zoo

from mxnet.gluon.model_zoo.vision import resnet50_v1
pretrained_net = resnet50_v1(pretrained=True, prefix = 'headpose_')
print(pretrained_net)

The ImageNet pre-trained model has 1,000 categorical outputs. However in our case, we only need nine. Thus, we need to modify the number of output classes to match our labels.

Modify the ResNet 50 model from model zoo

net = resnet50_v1(classes=9, prefix='headpose_')
net.collect_params().initialize()
net.features = pretrained_net.features
from mxnet import init
#net.output.initialize(init.Xavier(rnd_type='gaussian', factor_type="in", magnitude=2)) # MXNet 1.1.0
net.initialize(init.Xavier(rnd_type='gaussian', factor_type="in", magnitude=2)) # MXNet 0.12.1
print(net)

Another ResNet-50 model called “net” is prepared. The “net” has nine class-outputs. Then, features from “pretrained_net” are passed onto the “net”.  Note that the “net” is a model network in the serialized format. We are going to fine-tune the model.

Train the model

In this section we show you two helper functions for the training.

Training helper functions

from mxnet import autograd
from mxnet import nd

# Accuracy Evaluation
def eval_acc(data_iter, net, ctx):
    acc = mx.metric.Accuracy()
    for i, (data, label) in enumerate(data_iter):
        data = data.as_in_context(ctx)
        label = label.as_in_context(ctx)

        output = net(data)
        pred = nd.argmax(output, axis=1)
        acc.update(preds=pred, labels=label)
    return acc.get()[1]

# Training Loop
def train_util(net, train_iter, validation_iter, loss_fn, trainer, ctx, epochs, batch_size):
    metric = mx.metric.create(['acc'])
    lst_val_acc = []
    lst_trn_acc = []
    best_accuracy = 0
    for epoch in range(epochs):
        for i, (data, label) in enumerate(train_iter):
            # ensure context            
            data = data.as_in_context(ctx)
            label = label.as_in_context(ctx)
            
            with autograd.record():
                output = net(data)
                loss = loss_fn(output, label)

            loss.backward()
            trainer.step(data.shape[0])

        train_acc = eval_acc(train_iter, net, ctx)
        validation_acc = eval_acc(validation_iter, net, ctx)

        lst_trn_acc += [train_acc]
        lst_val_acc += [validation_acc]
                
        ### Save checkpoint weights
        #
        # We save the model in the symbolic format (i.e. XXX.json and XXX.params)
        #
        
        # Export .json and .params files
        # chkpt-XX-symbol.json does not come with softmax layer at the end. 
        net.export('chkpt-{}'.format(epoch)) 
        # Overwrite .json with the one with softmax
        net_with_softmax = net(mx.sym.var('data'))
        net_with_softmax = mx.sym.SoftmaxOutput(data=net_with_softmax, name="softmax")
        net_with_softmax.save('chkpt-{}-symbol.json'.format(epoch)) 
        print("Epoch %s | training_acc %s | val_acc %s " % (epoch, train_acc, validation_acc))
        
        if validation_acc > best_accuracy:
            # A network with the best validation accuracy is returned.  
            net_best = net
            net_with_softmax_best = net_with_softmax
            best_accuracy = validation_acc
        
    return lst_trn_acc, lst_val_acc, net_best

The first helper method is for the accuracy evaluation during the training, and the other is a training loop. For the sake using this with AWS DeepLens, save the checkpoint model artifacts in the symbolic format (that is, .json and .params). Because “net” is in the serial format, we use the .export method to save a serial model in the symbolic model artifact. In addition, we want a softmax output layer at the end of our network. Thus, we used the Symbol API method to add a softmax output and .save method to overwrite the .json file.

If you want to save the serialized model in the serial format, you can simply use the .save_param method. Here is an example of how you save the serialized model weights and pass the weights to another serialized model.

Save the model in the serial (Gluon) format

'''
Save net (Gluon model) in the serial format -> Pass the weights to another serialized model. 
'''

net_gluon.save_params('net_gluon.params')
net_gluon2 = resnet50_v1(classes=9, prefix='headpose_')
net_gluon2.load_params('net_gluon.params', ctx=mx.cpu())

We now have all the tools necessary to train the model. Let’s start the fine-tuning.

Fine-tune the model

def train(net, ctx, batch_size=64, epochs=10, learning_rate=0.0005):
    # Making Gluon iterators
    train_iter = mx.gluon.data.DataLoader(mx.gluon.data.ArrayDataset((trn_im.astype(np.float32)-0.5) *2, np_trn_grid_cls),
                                                batch_size=batch_size, shuffle=True, last_batch='discard')
    test_iter = mx.gluon.data.DataLoader(mx.gluon.data.ArrayDataset((test_im.astype(np.float32)-0.5) *2 , np_test_grid_cls),
                                                batch_size=batch_size, shuffle=True, last_batch='discard')

    net.collect_params().reset_ctx(ctx)
    net.hybridize()
    
    loss = gluon.loss.SoftmaxCrossEntropyLoss()
    trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': learning_rate})
    
    return train_util(net, train_iter, test_iter, loss, trainer, ctx, epochs, batch_size)

The hybridize method allows us to save the serialized model (“net”) in the symbolic format.

Hopefully, the rest of the notebook is self-explanatory. After you successfully run the training, you will have .json and multiple model weights (.params) from multiple checkpoints. You have to hand-pick one .param that gives you the best validation accuracy. We trained the model on a p2.8xl ec2 instance running the AWS Deep Learning AMI Ubuntu version, and we achieved validation accuracy of ~80% with this dataset.

Train the ResNet-50 model using Amazon SageMaker (Python SDK) with Gluon

So far, we walked through the basics of how to train a CNN using Gluon in Python. The next step is to reproduce the same model training experience on the Amazon SageMaker Python SDK. Amazon SageMaker is a fully-managed service that enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale. To use our dataset and code, we’ll write a custom entry point Python script to run on Amazon SageMaker.

S3 bucket

Create an AmazonS3 bucket first if you don’t have one. In this example, we are going to name the S3 bucket “deeplens-sagemaker-0000” hosted in the N. Virginia (US East 1) AWS Region. (If you want to deploy your trained model artifacts straight into AWS DeepLens, the region must be N. Virginia (US East 1)). 

Inside the bucket, we have a folder named “headpose.” Inside the “headpose” folder, we have 4 sub-folders named “artifacts,” “customMXNetcodes,” “datasets,” and “testIMs.”

You are going to host the head-pose dataset you created earlier (HeadPoseData_trn_test_x15_py2.pkl) in the datasets folder.

That is it for the preparation.

Amazon SageMaker notebook

Now, you launch Amazon SageMaker. After you open Amazon SageMaker notebook, upload our sample notebook and entry point Python script (HeadPose_SageMaker_PySDK-Gluon.ipynb and EntryPt-headpose-Gluon.py, respectively).

After you place the notebook and entry point script, there are only three steps for you to run the training.

First, specify your S3 bucket name in the sample Amazon SageMaker notebook (HeadPose_SageMaker_PySDK-Gluon.ipynb). In this part, you also specify other folders inside your S3 bucket such as the “headpose” folder as well as the “artifacts” and “customMXNetcodes” folders underneath it.

from sagemaker import get_execution_role

s3_bucket = ' deeplens-sagemaker-0000'
headpose_folder = 'headpose'

#Bucket location to save your custom code in tar.gz format.
custom_code_upload_location = 's3://{}/{}/customMXNetcodes'.format(s3_bucket, headpose_folder)

#Bucket location where results of model training are saved.
model_artifacts_location = 's3://{}/{}/artifacts'.format(s3_bucket, headpose_folder)

#IAM execution role that gives SageMaker access to resources in your AWS account.
#We can use the SageMaker Python SDK to get the role from our notebook environment. 
role = get_execution_role()

Second, specify the training instance and other parameters in the MXNet object. In this example, we use a ml.p2.xlarge instance for the training.

from sagemaker.mxnet import MXNet

headpose_estimator = MXNet(entry_point='EntryPt-headpose-Gluon.py',
                        role=role,
                        output_path=model_artifacts_location,
                        code_location=custom_code_upload_location,
                        train_instance_count=1, 
                        train_instance_type='ml.p2.xlarge',
                        hyperparameters={'learning_rate': 0.0005},
                        train_volume_size=100)

“train_max_run” represents the maximum training time that the training instance is running in units of seconds (432000 seconds = 5 days) in case that the training takes a long time. “train_volume_size” corresponds to the disk volume of the training instance in GB.

You also see that the MXNet object, headpose_estimator takes the name of entry point script (i.e. EntryPt-headpose-Gluon.py) as well as folder locations such as “model_artifacts_location” and “custom_code_upload_location”.

headpose_estimator.base_job_name = 'deeplens-sagemaker-headpose'

We name this job “deeplens-sagemaker-headpose”. The base_job_name will be the prefix of output folders we are going to create. For the model development for AWS DeepLens, it is a good practice to include both “deeplens” and “sagemaker” in the name of the Amazon S3 bucket as well as the name of the job.

Finally, we run the training by calling the “.fit” method. The method “.fit” takes the location of input dataset in the Python dictionary form.

%%time
import boto3

region = boto3.Session().region_name

'''
# Load preprocessed data and run the training#
'''

# Head-pose dataset "HeadPoseData_trn_test_x15_py2.pkl" is in the following S3 folder. 
dataset_location = 's3://{}/{}/datasets'.format(s3_bucket, headpose_folder)

# You can specify multiple input file directories (i.e. channel_input_dirs) in the dictionary.
# e.g. {'dataset1': dataset1_location, 'dataset2': dataset2_location, 'dataset3': dataset3_location}
# Start training !
headpose_estimator.fit({'dataset': dataset_location})

This is how we run the training using Amazon SageMaker Python SDK. You can monitor the progress of the training on either the Amazon SageMaker or the Amazon CloudWatch consoles.

Entry point Python script

All details for the head-pose model training are described in the entry point Python script (EntryPt-headpose-Gluon.py). You may want to closely look at the similarity between EntryPt-headpose-Gluon.py and HeadPose_ResNet50_Tutorial-Gluon.ipynb that we just discussed earlier. They are basically the same except for some instructions on directories to output and save model artifacts (such as model_dir and output_data_dir).

You may also want to compare EntryPt-headpose-Gluon.py and EntryPt-headpose.py, 

which is the symbolic Apache MXNet version of the head-pose entry point script. The notable difference between two entry point Python scripts is that the Gluon script has two additional functions (save and model_fn).

def save(net, model_dir):
    '''
    Save the model in the symbolic format. 
    
    :net: serialized model returned from train
    :model_dir: model_dir The directory where model files are stored.
    
    DeepLens requires a model artifact in the symbolic format (.json and .params)
    
    '''
    
    ### Save modularized model 
    # Export .json and .params files
    # model-symbol.json does not come with softmax layer at the end. 
    net.export('{}/model'.format(model_dir)) 
    # Overwrite model-symbol.json with the one with softmax
    net_with_softmax = net(mx.sym.var('data'))
    net_with_softmax = mx.sym.SoftmaxOutput(data=net_with_softmax, name="softmax")
    net_with_softmax.save('{}/model-symbol.json'.format(model_dir))

Because we want to develop the model for AWS DeepLens, we need to save the model in the symbolic format. However, the “net” in the function “train” is in the serial Gluon format. The function “save” accepts the return from function “train”, saves the format in any format that you want and places the model artifact in model.tar.gz. Without the “save”, Amazon SageMaker automatically saves the “net” in its default format.

def model_fn(model_dir):

    """
    Load the model. Called once when hosting service starts.

    :param: model_dir The directory where model files are stored.
    :return: a model 
    """
    
    model_symbol = '{}/model-symbol.json'.format(model_dir)
    model_params = '{}/model-0000.params'.format(model_dir)

    sym, arg_params, aux_params = load_model(model_symbol, model_params)
    ### DSHAPR = (1,3,84,84)
    # The shape of input image. 
    dshape = [('data', DSHAPE)]

    ctx = mx.cpu() # USE CPU to predict... 
    net = mx.mod.Module(symbol=sym,context=ctx)
    net.bind(for_training=False, data_shapes=dshape)
    net.set_params(arg_params, aux_params)
    
    return net

The function “model_fn” is served for the same reason—the discrepancy between the formats of trained network and saved model artifact for the inference hosting.

After you successfully run the training, you have model.tar.gz in output folder inside artifacts folder. Inside the model.tar.gz, you find a pair of model-symbol.json and model-0000.params that produced the best validation accuracy during the training.

If you have your AWS DeepLens account, you can find the model artifacts you just developed on AWS DeepLens console.

In the AWS DeepLens console, choose Models, select “Amazon SageMaker trained model” and scroll down the Job IDs. You can immediately deploy the model to your AWS DeepLens device.

Conclusion

In this blog post we developed a head-pose estimator CNN model using the Gluon interface in Apache MXNet using Amazon SageMaker and deployed on to the AWS DeepLens device. We also dove deep into the difference between symbolic and serial model formats and showed you how to handle them for your own application. The symbolic interface for the same application is also provided in the github repo (HeadPose_ResNet50_Tutorial.ipynb, HeadPose_SageMaker_PySDK.ipynb, and EntryPt-headpose.py).

The Amazon SageMaker Python SDK allows you to bring your custom Apache MXNet or Gluon script and dataset and makes it easy to train, deploy, and test Deep Learning models.

About the Authors

Tatsuya Arai PhD is a biomedical engineer turned deep learning oriented data scientist at Amazon ML Solutions Lab. He believes that the power of AI isn’t exclusively for computer scientists or mathematicians.

Vikram Madan is a Senior Product Manager for AWS Deep Learning. He works on products that make deep learning engines easier to use with a specific focus on the open source Apache MXNet engine. In his spare time, he enjoys running long distances and watching documentaries.

Brad Kenstler is a Data Scientist on the AWS Deep Learning Team. As part of the AWS ML Solutions Lab, he helps customers adopt ML & AI within their own organization through educational workshops and custom modeling. Outside of work, Brad enjoys listening to heavy metal and bourbon tasting.

Sunil Mallya is a Senior Solutions Architect in the AWS Deep Learning team. He helps our customers build machine learning and deep learning solutions to advance their businesses. In his spare time, he enjoys cooking, sailing and building self driving RC autonomous cars.

Jyothi Nookula is a Senior Product Manager for AWS DeepLens. She loves to build products that delight her customers. In her spare time, she loves to paint and host charity fund raisers for her art exhibitions.