Train and host Scikit-Learn models in Amazon SageMaker by building a Scikit Docker container

It is especially important to know the correct training input location (/opt/ml/input/data/) and model location (/opt/ml/model/) for your program because it aids in local testing and debugging. Detailed mounting location mapping can be found in Using Your Own Training Algorithms, and Using Your Own Inference Code.

Building in practice with Scikit-Learn

“Build” is the first step of any Amazon SageMaker workflow. For Scikit in particular, this involves building three things: 1) the model, 2) the training code, and 3) the inference code. This work will need to be containerized within a Docker image.

Build the training code for Docker

When bringing your own model, like we are with our Scikit model, we need to build our model inside of a Python file, instead of inside of a Jupyter notebook. You can still experiment with your model pipeline in the notebook instance, but once you’re ready to move to the training cluster, we need the code in its own file. This is so that we can containerize it with Docker.

Let’s look at the sample training file that comes with the example notebooks, called train. This is the code that is launched when Amazon SageMaker launches a training job with a container. This file may be a bit overwhelming, so let us look at the important sections.

First, lines 21-31 define all the paths to where SageMaker mounts interesting things below in container’s /opt/ml/:

• training data, including channels
• configuration information,
• model
• output

Then, define a function to execute the training. If you are primarily concerned with building a Scikit model on Amazon SageMaker, the first important lines are 52-53:

train_y = train_data.ix[:,0]
train_X = train_data.ix[:,1:]

These lines specify what columns in your data are your label columns. In this sample, we use the Iris dataset. The first column of the CSV file is our target label, so we’re assigning that to the train_y variable. The remaining columns are then assigned to train_X. It’s also important to note that the Iris data doesn’t have a header column. For your data, you either have to edit these lines of code in train, or modify your incoming data file to look similar to the Iris data.

The next block of code has to do with obtaining any hyperparameters that may have been passed as part of the training job.  This is in lines 57-59:

max_leaf_nodes = trainingParams.get('max_leaf_nodes', None)
if max_leaf_nodes is not None:
    max_leaf_nodes = int(max_leaf_nodes)

In the example train file, there is only a single parameter passed, but it shows how to get the raw string that’s passed, and how to convert the string to the proper format. In this case, we want the max_leaf_nodes parameter as an integer. This can be expanded for the number of hyperparameters that you want to set for your own model.

Next, and probably most importantly, we define our Scikit model and call fit. This happens on lines 62 and 63:

clf = tree.DecisionTreeClassifier(max_leaf_nodes=max_leaf_nodes)
clf = clf.fit(train_X, train_y)

This code should look very familiar if you’ve worked with Scikit before. Aside from how we’re setting up the hyperparameters, the model definition and the model fit is identical to how this would be done in a notebook. For the hyperparameters, notice that we’re pulling the max_leaf_nodes variable obtained above in the hyperparameter section. You can set this model definition to whatever Scikit model is most appropriate for your data.

Finally, see how the model is saved by the train file. This can be seen in lines 66-67:

with open(os.path.join(model_path, 'decision-tree-model.pkl'), 'w') as out:
    pickle.dump(clf, out)

We are using pickle for our object serialization, to save this model. It will be placed in the output folder of the docker container, and eventually pushed to an Amazon S3 location. These lines are the only lines in train that you really need to be concerned with if you want to build your own Scikit model in Amazon SageMaker.

Build the Inference Code for Docker

In addition to the train code that is run during a training job, we also want to add inference code to our container. This code is contained in predictor.py. This is the code that gets called when you try to connect to a SageMaker endpoint.

To start, we need to make sure we’re loading our model. We can see this on lines 30-31:

with open(os.path.join(model_path, 'decision-tree-model.pkl'), 'r') as inp:
    cls.model = pickle.load(inp)

The important thing here is to make sure that you have the right name for your pickle file. We saw just earlier that we named this file decision-tree-model.pkl in the train code, so we want to make sure that is the same here.

The next important lines are the actual predict lines, on 42-43:

        clf = cls.get_model()
        return clf.predict(input)

We don’t need to make any changes to this code, but it’s important to know this is how the actual prediction function gets called for your Scikit model.

Finally, we’ll want to take a look at how the model gets the input data. This happens in lines 67-69:

data = flask.request.data.decode('utf-8')
s = StringIO.StringIO(data)
data = pd.read_csv(s, header=None)

These are probably the most important lines to learn because the loading of data is different than most data scientists are used to in their normal workflow. The data here is acquired from the Flask framework, which we treat as a file-like object in memory (StringIO), which we then load into Pandas using the read_csv function. The data itself is CSV data passed along to Flask by an http POST method.

We then get the prediction results from this line 76:

predictions = ScoringService.predict(data)

Finally, we return our prediction results by turning them back into a CSV file, using the following lines (79-83):

out = StringIO.StringIO()
pd.DataFrame({'results':predictions}).to_csv(out, header=False, index=False)
result = out.getvalue()

return flask.Response(response=result, status=200, mimetype='text/csv')

As you can see, you should not need to make many changes to predictor.py to get your model to deploy effectively. Make sure the input data is formatted properly, and that the same pickle file is being loaded that you saved from training.

Build the container

Now that we have our code in train and predictor.py, we can build our Docker container and push it to Amazon ECR. Create a directory with the following files:

decision_trees/
    nginx.conf
    predictor.py
    serve
    train
    wsgi.py

This is the program folder we discussed in the overview. Again, we only have to pay attention to predictor.py and train. The rest we can pull without editing from the SageMaker sample notebooks as-is. Details can be found here.

container/
    decision_trees/
    Dockerfile

We also need a file to tell Docker how to build the image, known as Dockerfile. This goes in the parent container folder of the program folder we created earlier. You don’t need to make any changes to this file, except the name of the program folder, which is decision_trees in the example. You can download it from the sample SageMaker notebooks.

In addition, we provide a helper script for building the Docker image, called build_and_push.sh. Now that we have all the pieces, we can run this script.

To practice the build and push process, we will reuse scripts in the sample-notebooks that come with Amazon SageMaker notebook instances, and we’ll run them on a notebook instance. Once you grasp the essentials of the building process, you can choose to replicate the process and build your own helper scripts from any terminal using the AWS Command Line Interface (CLI) installed and configured. Here is how you start a terminal on an opened notebook instance.

After you start the terminal, the example container folder can be located in the following container folder:

cd ~/SageMaker/sample-notebooks/advanced_functionality/ \
scikit_bring_your_own/container
ls
ls decision_trees/

The build_and_push.sh script will acquire the Region value from the current AWS CLI configuration. If no Region is currently set, it defaults to us-west-2. You can change the Region with the following command:

aws configure set region us-east-1

For training, you want your Amazon S3 bucket for training data to be in the same region as your Amazon SageMaker training job and your training container. For inference, you will need to push your model and your inference container to each Region in which you want to have it available in Amazon SageMaker. For example, if you are planning to deploy SageMaker endpoints in multiple Regions to reduce latency, it makes sense to deploy the container to multiple Regions as well.

To make sure files inside the program folder are set as executable, run the following commands before the build:

chmod +x decision_trees/train
chmod +x decision_trees/serve

The build_and_push.sh also needs to be set to executable, and requires an argument for a name of the image when running the script. Here we name the image decision_trees, and the Docker image will be created and pushed to Amazon ECR.

chmod +x build_and_push.sh
./build_and_push.sh decision_trees

You will see output similar to this screenshot if your build and push is successful. The push refers to a repository name with the following structure:

<account number>.dkr.ecr.<region>.amazonaws.com/<image name>

You can also see our decision-trees image inside of Amazon ECR. Open the AWS Management Console, choose Services, and then Elastic Container Service under Compute. Select Repositories on the left, and you will see the decision-trees Repository name. Choose it, so you can see the Repository URI. This should match the terminal output of running build_and_push.sh. Save this for later.

Testing and debugging

The build and push are combined in one step by running the build_and_push.sh script. But logically, before pushing the image to Amazon ECR, run some local tests to insure the Docker image functions the way you anticipate. Assume that you have the Docker image that you want to test built at this moment. You should see a repository decision_trees by listing locally built Docker image by running the following:

docker image ls

AWS provides sample scripts for running local tests of your container prior to creating any jobs or endpoints on Amazon SageMaker. They are: train_local.sh, serve_local.sh, and predict.sh, which can also be found in an example local_test folder. You can download the local_test folder anywhere with the same folder structure that follows as the example local_test folder. The test_dir/ will be mounted from your operating system to /opt/ml/ inside the Docker container.

local_test/
    test_dir/
        input/
            config/
                hyperparameters.json
                resourceConfig.json
            data/
                training/
                    iris.csv
        model/
        output/
    payload.csv
    predict.sh
    serve_local.sh
    train_local.sh

To run the sample scripts for testing and debugging on the SageMaker notebook instance, let’s go back to the sample local_test folder.

cd ~/SageMaker/sample-notebooks/advanced_functionality/scikit_bring_your_own/container/local_test

Note, you might first need to set the executable flag for these files with the following command:

chmod +x *.sh

Testing the container for training

train_local.sh will start a training job on your local built Docker image, using a sample data iris.csv. in ./test_dir/input/data/training/. You call it with the following command:

./train_local.sh decision_trees

You can run local testing for training with an empty JSON for both havehyperparameters.json and resourceConfig.json, and you need to have a training channel under data/ that contains the training data. These are all aligned with the logic in the training code. The structure and contents under test_dir/input/ should be adjusted to your own training code.

We highly recommend that you use a very small sample of your data for local testing, and use the full data for the actual training job. If the script is successful, you should see a generated model file in the following directory ./test_dir/model/.

Now let’s try debugging with a simulation of a failed run, by deleting the hyperparameters.json file under ./test_dir/input/config/. The failed run will generate a output file failure under ./test_dir/output/ logging the details for your debugging and troubleshooting.

Testing the container for hosting

If training runs smoothly, you’ll also want to test the inference code. First, make sure there is a model file under ./test_dir/model/, then, start the RESTful service locally, by running the following script:

./serve_local.sh <image name> > output.log

Any output of our Flask application will now be sent to the ./output.log file.

Finally, we can test the endpoint code with the following command. You need to start another terminal:

./predict <payload file> [content-type]

The content-type is optional, and will default to text/csv. Now check the ./output.log file to make sure that the payload was accepted properly.

If all these scripts run well, then you’re all set to deploy a training job on Amazon SageMaker, and to later deploy the container in an endpoint! We have one last step to clean up our development environment. We need to stop our RESTful service. If this is the only Docker container you have running, you can stop it with the following command:

docker ps -a
docker kill <CONTAINER ID>

If you are running other Docker processes on your instance, you will want to locate each specific process and kill it.

Train and deploy using the Amazon SageMaker console

Create a training job

Now that we have our Docker image, and have set up our training and inference code, we can run a training job. Go to the AWS Management Console, choose Services, and then Amazon SageMaker under Machine Learning. Select Jobs on the left, and then choose the Create training job button.

You’ll see the Create training job window. Set a Job name like decision-trees-<datetime>. Adding the datetime is recommended to avoid ResourceInUse in job name. Under IAM role choose Create a new role, or Enter a custom IAM role ARN, which can be any role that has the AmazonSageMakerFullAccess, and AmazonS3FullAccess policy attached. Set your Algorithm to Custom. In our example, we use File as the Input mode. Next, in the Training image cell, put in your Docker Repository URI, and the Image Tag (the build_and_push.sh tag the Docker image as latest).

Scroll down to the Resource configuration. These configurations can be left alone for our example, but feel free to customize them for your own models. The main configuration here is the Instance type, as well as the number of instances under Instance count. Note, some algorithms, like Scikit algorithms, can only take advantage of a single instance. Setting this higher won’t automatically cause your algorithm to scale across multiple instances.

Your primary choices for instance type are from the m, c, and p families of instances. The m family is a balanced instance type, that is a good place to start if you are uncertain about where your model might be most intensive. The c family is for compute intensive workloads, while the p family provides GPU capacity, often used for deep learning. You can better see where your model is most intensive by monitoring CloudWatch for SageMaker.

Set any hyperparameters for this job in the Hyperparameters box. In our example, the only hyperparameter we can set is max_leaf_nodes. You may optionally set a value for this here. If you are interested in enabling more hyperparameter options for your algorithm, remember these need to be passed in your train file in the Docker image.

Next, set Input data configuration. The Channel name has to align with what you programmed in your train script. We recommend that the S3 location that you set up for your data input matches the channel name, especially when you have multiple channels. This way Amazon SageMaker can inject channel data to the corresponding container location (e.g. /opt/ml/input/data/<channel_name>), in the location where program train looks for data. In our example, we have a single channel called training. Our model is expecting a CSV file, with no header, where the first column is our target label. Feel free to upload the famous Iris data set to your own S3 location for this, as it is what is used in this example. Make sure to choose Done.

In our example, we will have the training data folder as follows.

Finally, you need to specify where the model artifact is going to be saved on Amazon S3 after the training job is done. Remember, this takes anything in the container that is output to /opt/ml/model/, and then it archives it with tar and gzip compresses it. Specify an Amazon S3 bucket in the S3 output path field.

Next, choose the Create training job button.

After the created training job is completed, you should be able to see the generated model in the S3 output path you provided. The full path of the model is

s3://<bucket name>/models/<job name>/output/model.tar.gz

Deploy an endpoint

After the training job has finished, we can deploy an endpoint, to take our model to production. We can do this from the SageMaker console. There are three steps: First, we have to create a model resource. Second, we create an endpoint configuration. Third, we deploy the actual endpoint.

The first step is to create the model resource. This contains the information Amazon SageMaker needs about the Docker image and the Amazon S3 location of the artifact.

In the Amazon SageMaker console, in the left navigation pane, under Resources select Models. Then choose the Create model button.

First we need to input a Model name, with a name like decision-trees, and then select or create an IAM role. You can use the same role that you used for your training job, as long as it has the AmazonSageMakerFullAccess IAM policy attached.

Next, we need to provide information about our container. We can use the same container we used for the training job. Provide the ECR URI that we saved earlier for our container in the Location of inference code image field. Make sure to include an image tag, like the latest tag. Next, in the Location of model artifacts cell, provide the S3 URI for the model artifact that you created in the training job. This was the S3 output path you set during training.

Choose the Create model button.

Next, we need to create an Endpoint configuration. We combine this with model resources to determine what kind of instances to provision. You can also use this to specify more than one model for doing A/B testing. For our example, we’ll only use a single model.

On the left navigation pane, under Resources, select Endpoint configuration. Choose the Create endpoint configuration button.

Give your endpoint configuration a name under Endpoint configuration name. Unless we anticipate having multiple versions of the same model, provide the same name as my model. Next, choose the Add model link in the lower left. Select the model you just created, and choose save. You can leave the additional values as their defaults for this example.

Choose the Create endpoint configuration button.

Now that we have a model and an endpoint configuration, we can deploy an actual endpoint. Activating an endpoint will provision an instance. Note that as long as the endpoint is active, charges will accrue, so it’s important to only activate an endpoint when you intend to direct traffic towards it.

In the left navigation pane, under Resources, select Endpoints. Choose the Create endpoint button.

Give your endpoint a name under Endpoint name. Use the same name as my model when we are only deploying a single model with no variants. In this case, use decision-trees. Leave Use an existing endpoint configuration selected.

Under Endpoint configuration, select the endpoint configuration we created in the last step. Then choose the Select endpoint configuration button. Finally, choose the Create endpoint button.

The result should look something like this:

Train and deploy using the AWS CLI

Create a training job

You have the option to launch a training job from the AWS CLI with the following command:

aws sagemaker create-training-job \
    --training-job-name <job name> \
    --algorithm-specification \
        TrainingImage=<Repository URI>:<Image Tag>,TrainingInputMode=File \
    --role-arn <Role ARN> \
    --input-data-config \
        '{"ChannelName":"training", 
          "DataSource":{"S3DataSource":{"S3DataType":"S3Prefix", 
                                        "S3Uri":"s3://<bucket name>/data/training/",
                                        "S3DataDistributionType":"FullyReplicated"
                                        } 
                        } 
          }' \
    --output-data-config \
        S3OutputPath=s3://<bucket name>/models/ \
    --resource-config \
        InstanceType=ml.m4.xlarge,InstanceCount=1,VolumeSizeInGB=1 \
    --stopping-condition \
        MaxRuntimeInSeconds=86400 \
    --hyper-parameters \
        max_leaf_nodes=5 

This command will create the same training job with same configuration as the one we created in the Amazon SageMaker console, except for the name of the training job because it has to be a unique identifier. The returned TrainingJobArn indicates that the creation of the training job is successful.  You can track the status of the training job in the Amazon SageMaker console, on the Jobs page.

The input-data-config allows an array of channel objects. For example, you can provide another channel if defined, by attaching it after the training channel, and using space to separate these two objects. The hyper-parameters allows a maximum of 100 hyperparameters, such as KeyName1=string,KeyName2=string. For example, if we have max_depth defined, we can attach max_depth=10 right after max_leaf_nodes=5. Please check aws.sagemaker.create-training-job and Using Shorthand Syntax with the AWS Command Line Interface for more detail when creating your own training job using AWS CLI.

Also, you can see that the S3OutputPath contains both training jobs now created from the Amazon SageMaker console and the AWS CLI.

Deploy an endpoint

You can use the following three AWS CLI commands to deploy a SageMaker endpoint. We also provide the expected output for the commands.

aws sagemaker create-model \
    --model-name <model name> \
    --primary-container \
        Image=<Repository URL>:<Image Tag>,ModelDataUrl=s3://<bucket name>/models/<job name>/output/model.tar.gz \
    --execution-role-arn <Role ARN>

aws sagemaker create-endpoint-config \
    --endpoint-config-name <endpoint config name> \
    --production-variants \
        VariantName=dev,ModelName=<model name>,InitialInstanceCount=1,InstanceType=ml.m4.xlarge,InitialVariantWeight=1.0

aws sagemaker create-endpoint \
    --endpoint-name <endpoint name> \
    --endpoint-config-name <endpoint config name>

You should see your endpoint InService after it is created successfully.

Test inferences

The easiest way to test our endpoint is from the SageMaker notebook instance. In a cell, run the following code:

import boto3
import io
import pandas as pd
import itertools

# Set below parameters
bucket = '<bucket name>'
key = 'data/training/iris.csv'
endpointName = 'decision-trees'

# Pull our data from S3
s3 = boto3.client('s3')
f = s3.get_object(Bucket=bucket, Key=key)

# Make a dataframe
shape = pd.read_csv(io.BytesIO(f['Body'].read()), header=None)

# Take a random sample
a = [50*i for i in range(3)]
b = [40+i for i in range(10)]
indices = [i+j for i,j in itertools.product(a,b)]
test_data=shape.iloc[indices[:-1]]
test_X=test_data.iloc[:,1:]
test_y=test_data.iloc[:,0]

# Convert the dataframe to csv data
test_file = io.StringIO()
test_X.to_csv(test_file, header=None, index=None)

# Talk to SageMaker
client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(
    EndpointName=endpointName,
    Body=test_file.getvalue(),
    ContentType='text/csv',
    Accept='Accept'
)

print(response['Body'].read().decode('ascii'))

This will take a random sample of the data in your data file, and send it to the end point. It will also print out the result.

Conclusion

Congratulations! You have now built, trained, deployed, and tested your Scikit model in Amazon SageMaker. Feel free to modify any part of this workflow or code to better suit your own needs.

About the Authors

Morgan Du is a Machine Learning Consultant with AWS Professional Services. She is passionate about artificial intelligence, machine learning, and deep learning. She is dedicated in enabling innovative ML solutions that are impacting and powering our everyday life.

Thomas Hughes is a Data Scientist with AWS Professional Services. He has a PhD from UC Santa Barbara and has tackled problems in in the social sciences, education, and advertising. He is currently working to solve some of the trickiest problems that arise when machine learning meets big data.