Bites of IoT: Creating AWS IoT Rules with AWS CloudFormation

Welcome to another installment in the Bites of IoT blog series.

In this bite, we will use AWS CloudFormation, the AWS IoT rules engine, and AWS Lambda to automate the setup and teardown of two AWS IoT rules. You can use the AWS IoT console to create and edit rules by hand, but in production you might want to use automation to make your deployments of rules repeatable and easier to manage. AWS CloudFormation enables you to deploy rules consistently across applications, manage updates, share your infrastructure with others, and even use revision control to track changes made over time.

Configure the CLI

As with all Bites of IoT posts, we are using the AWS IoT ELF client available in AWS Labs on GitHub. If you aren’t familiar with the ELF, see the first post in this series.

What Are Rules?

AWS IoT rules are SQL statements that can be used to perform three kinds of functions on MQTT messages:

Test: A rule can test an MQTT message to determine if it meets some criteria. For example, a rule could check to see if a temperature field is above or below a threshold, or if a text field contains a certain string.
Transform: A rule can pass an MQTT message through without changing it or it can transform it in some way. There are several SQL functions to support transformations. For more information, see the AWS IoT SQL Reference. Common transformations include changing a value from one system of measurement to another (e.g. Fahrenheit to Celsius), hashing sensitive information to obscure it from downstream systems (e.g. MD2, MD5, SHA1, SHA224, SHA256, SHA384, SHA512), removing information when it isn’t useful in another data processing stage, or adding information required by other processes (e.g. timestamps).
Trigger: When a rule is evaluated and its test criteria (if any) is met, it triggers an action. The examples in this post cover the republish action and the AWS Lambda action. The republish action takes a message and republishes it to another topic. The AWS Lambda action sends the message to an AWS Lambda function. There are several other actions available that you can use to tie into other AWS services.

Creating a SQL-Only Rule with AWS CloudFormation

The first rule we are going to create will receive the “IoT ELF hello” message from ELF. It will republish a new message on an output topic that indicates which client sent the message. We will create the rule using the SQL rules engine in AWS IoT.

Here is the SQL statement for our first rule:

SELECT concat(topic(2), " says hello!") AS feedback FROM 'input/#'

Let’s walk through each part to see what it does:

SELECT – All SQL statements in the rules engine start with SELECT.
concat(topic(2), " says hello!") – The concat function combines two strings.
- The first string is obtained from the topic(2) function, which means it uses the second segment of the topic. Because ELF publishes messages on input/thing_X where X is the thing’s ID, this string will be either thing_0 or thing_1.
- The second string is simply says hello!. This part of the statement will evaluate to either thing_0 says hello! or thing_1 says hello!
AS feedback – This means that, in our output message, the string we just constructed will be referred to as feedback. Our full output message will be either { "feedback": "thing_0 says hello!" } or { "feedback": "thing_1 says hello!" }.
FROM 'input/#' – This means that we want this rule to receive messages on any topic under the topic input. This would match input/thing_0, input/thing_1, or even input/thing_0/one_more_level. If we didn’t want to match all topics under input, we could change input/# to input/+. That would only match input followed by one additional level in the topic hierarchy. We wouldn’t process messages on the topic input/thing_0/one_more_level if we were to use input/+.

YAML AWS CloudFormation Template for SQL-Only Rule

AWSTemplateFormatVersion: 2010-09-09
Description: A SQL only IoT republish rule that responds to a device saying hello

Resources:
  SQLRepublishTopicRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: SQLRepublish
      TopicRulePayload:
        RuleDisabled: false
        Sql: SELECT concat(topic(2), " says hello!") AS feedback FROM 'input/#'
        Actions:
          - Republish:
              Topic: output/${topic(2)}
              RoleArn: !GetAtt SQLRepublishRole.Arn
  SQLRepublishRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - iot.amazonaws.com
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic/output/*" ] ]

YAML AWS CloudFormation Template Explained

Let’s break this down into pieces.

SQLRepublishTopicRule Section

  SQLRepublishTopicRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: SQLRepublish
      TopicRulePayload:
        RuleDisabled: false
        Sql: SELECT concat(topic(2), " says hello!") AS feedback FROM 'input/#'
        Actions:
          - Republish:
              Topic: output/${topic(2)}
              RoleArn: !GetAtt SQLRepublishRole.Arn

This section creates an AWS resource that is a topic rule (AWS::IoT::TopicRule) with several properties.

The first property is the rule name SQLRepublish. You’ll see this rule name in the AWS IoT console after this template has been launched.
The second property is the topic rule payload. The topic rule payload contains several attributes:
The first attribute indicates that the rule is enabled. Rules can be disabled so they aren’t executed, but remain in AWS IoT console. That way, you can easily enable them rather than creating them again.
The second attribute is the SQL statement we explained earlier.
The third attribute contains the actions performed when the rule’s criteria is met. In this case, we want to republish to an output topic using an IAM role specified in the next section of the AWS CloudFormation template. We specify the role using the role’s ARN attribute. !GetAtt is the YAML syntax’s get attribute intrinsic function. All functions in YAML templates are prefixed with an exclamation point !.

There are other intrinsic functions that are useful for managing your stack. They provide variables that are available during runtime only.

The topic here is output/${topic(2)}. This syntax means that it will extract the second segment of the input topic (e.g. thing_0 from input/thing_0) and use it in that location. If we received an input message from thing_0, our output topic would be output/thing_0. This syntax allows the rule to dynamically publish to any of a number of output topics without a separate rule for each thing or each input topic.

SQLRepublishRole Section

  SQLRepublishRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - iot.amazonaws.com
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic/output/*" ] ]

This section creates an AWS resource that is an IAM role (AWS::IAM::Role) with several properties:

The first property is the assume role policy document. This property allows AWS IoT to assume this role in your account so it can republish messages from the rules engine.
The second property is the list of policies associated with this role.

This role has one policy assigned to it called root, but the name is unimportant. It can be changed to something else, if you like. The policy contains one statement that has three attributes:

The first attribute indicates that the effect of this statement is to allow access to a particular action.
The second attribute declares the action. In this case, it is the iot:Publish action that lets an application publish an MQTT message.
The third attribute is a resource that we’ll break down in sections.

Here is the full resource statement:

Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic/output/*" ] ]

The !Join function joins together a list of strings. It includes a separator string between each pair of strings. In our case, we don’t want to add strings, so we specified an empty string ("") as the separator. This means the strings will be joined exactly as they are specified.

The next few statements build the ARN of the topic that we want to republish to. AWS IoT topic ARNs look like this:

arn:aws:iot:<region>:<accountId>:topic/<topicName>

The beginning of the ARN is always arn:aws:iot: followed by the region, a colon, the current account ID, a colon, the string topic/, and the topic name. The topic name can include wildcards because it is an IAM resource, but the use of the MQTT wildcards # and + are not allowed. You can only use * and ?.

To make this code reusable across accounts and regions, we use !Ref "AWS::Region" and !Ref "AWS::AccountId" to fill in the region and account ID automatically. The !Ref function tells AWS CloudFormation to handle this for us.

Deploying the SQL-Only Rule

Here are the steps for deploying the rule:

Save the entire template to a file named sql-hello.yaml.
Sign in to the AWS Management Console, and then open the AWS CloudFormation console.
Choose Create Stack.
Under Choose a template, click Choose file, select sql-hello.yaml, and then choose Next.
For stack name, type SQLRule, and then choose Next.
On the Options page, leave the fields at their defaults, and then choose Next.
On the Review page, select I acknowledge that AWS CloudFormation might create IAM resources, and then choose Create.

The state displayed for your AWS CloudFormation stack should be CREATE_IN_PROGRESS. You can periodically click the circular arrow in the upper-right corner to refresh the view. When the stack has been created, the state displayed for your stack will be CREATE_COMPLETE.

Testing the SQL-Only Rule

We can now use the ELF client to test the rule. If your client hasn’t been cleaned up since the last time you used it, you must first execute this command:

python elf.py clean

Now open two terminals. One terminal will publish messages. The other will subscribe to the messages coming from our rule.

In the first terminal, execute these commands:

python elf.py create 2
python elf.py send --topic input --append-thing-name --duration 60

In the second terminal, execute this command:

python elf.py subscribe --topic output --append-thing-name --duration 60

In the first terminal, you should see messages like this:

INFO - ELF thing_0 posted a 24 bytes message: {"msg": "IoT ELF Hello"} on topic: input/thing_0
INFO - ELF thing_1 posted a 24 bytes message: {"msg": "IoT ELF Hello"} on topic: input/thing_1

In the second terminal, you should see messages like this:

INFO - Received message: {"feedback":"thing_0 says hello!"} from topic: output/thing_0
INFO - Received message: {"feedback":"thing_1 says hello!"} from topic: output/thing_1

If you see messages in both terminals, then everything is working. Now you can go back to the AWS CloudFormation console, choose your SQLRule stack, and then choose Delete Stack. Wait until the stack has been deleted, and then try the commands again in the two terminals. You should see the same messages in the first terminal, but no messages in the second.

Creating a Rule with AWS CloudFormation to Route Messages to AWS Lambda

Now we’ll create a rule that routes MQTT messages to AWS Lambda. AWS Lambda will receive the message and publish a new MQTT message based on what it receives from the rules engine.

YAML AWS CloudFormation Template for AWS Lambda Rule

AWSTemplateFormatVersion: 2010-09-09
Description: A simple IoT republish rule that responds to a device saying hello with AWS Lambda

Resources:
  LambdaRepublishTopicRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: LambdaRepublish
      TopicRulePayload:
        RuleDisabled: false
        Sql: SELECT topic(2) AS thing_name, msg FROM 'input/#'
        Actions:
          - Lambda:
              FunctionArn: !GetAtt LambdaHelloFunction.Arn
  LambdaRepublishRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: logs:*
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic/output/*" ] ]
  LambdaHelloFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: LambdaHello
      Role: !GetAtt LambdaRepublishRole.Arn
      Timeout: 5
      Handler: index.lambda_handler
      Runtime: python2.7
      MemorySize: 512
      Code:
        ZipFile: |
                  import boto3
                  import json

                  def lambda_handler(event, context):
                      client = boto3.client('iot-data')
                      thing_name = event['thing_name']
                      payload = {}
                      payload['feedback'] = thing_name + " said hello to Lambda!"
                      payload = bytearray(json.dumps(payload))
                      response = client.publish(topic='output/' + thing_name, qos=0, payload=payload)
  LambdaInvocationPermission:
    Type: AWS::Lambda::Permission
    Properties:
      SourceArn: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":rule/", !Ref "LambdaRepublishTopicRule" ] ]
      Action: lambda:InvokeFunction
      Principal: iot.amazonaws.com
      FunctionName: !GetAtt LambdaHelloFunction.Arn
      SourceAccount: !Ref AWS::AccountId

YAML AWS CloudFormation Template Explained

Although this template is similar to the SQL-only template, there are some important differences.

The first is the SQL statement, which looks like this:

SELECT topic(2) AS thing_name, msg FROM 'input/#'

We’re using SQL to extract the topic(2) value to a field called thing_name and we’re passing through the msg field. This creates a JSON message with two fields that will be sent to our Lambda function. When we were using the republishing feature of the rules engine, we could access this value directly and use it to specify our output topic. When Lambda receives the JSON message from AWS IoT, the information it needs for its processing must be included in the message.

LambdaRepublishRole Section

  LambdaRepublishRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: logs:*
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic/output/*" ] ]

This section creates a role that Lambda can assume. It allows Lambda to perform the following actions in our account:

Write to Amazon CloudWatch Logs with logs:* and arn:aws:logs:*:*:*
Publish to the output topic hierarchy as we did in the other template

AWS IoT no longer needs publish permission because Lambda will handle publishing the messages for us.

LambdaHelloFunction Section

  LambdaHelloFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: LambdaHello
      Role: !GetAtt LambdaRepublishRole.Arn
      Timeout: 5
      Handler: index.lambda_handler
      Runtime: python2.7
      MemorySize: 512
      Code:
        ZipFile: |
                  import boto3
                  import json

                  def lambda_handler(event, context):
                      client = boto3.client('iot-data')
                      thing_name = event['thing_name']
                      payload = {}
                      payload['feedback'] = thing_name + " said hello to Lambda!"
                      payload = bytearray(json.dumps(payload))
                      response = client.publish(topic='output/' + thing_name, qos=0, payload=payload)

This section defines our Lambda function: It’s written in Python, gets 512 MB of RAM, has a five-second timeout, uses the role we just defined to publish messages in AWS IoT, and its code is specified inline.

The code creates an IoT data client with Boto 3, extracts the thing name, builds a payload in which the feedback field is populated with our message, converts the payload dictionary to JSON, converts the JSON to a byte array, and then publishes it to the correct output topic by appending the thing name to output/. The IoTDataPlane publish function in Boto 3 requires that the data passed to it is either a byte array or a reference to a file.

LambdaInvocationPermission Section

  LambdaInvocationPermission:
    Type: AWS::Lambda::Permission
    Properties:
      SourceArn: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":rule/", !Ref "LambdaRepublishTopicRule" ] ]
      Action: lambda:InvokeFunction
      Principal: iot.amazonaws.com
      FunctionName: !GetAtt LambdaHelloFunction.Arn
      SourceAccount: !Ref AWS::AccountId

This final section is different from the SQL republish template. This permission allows the Lambda function to be invoked by the AWS IoT rule. Without this permission, even if the rule is executed, Lambda will not allow AWS IoT to run the code.

Deploying and Testing the AWS Lambda Rule

Save this template as lambda-hello.yaml, launch it the same way we launched the last template but use the name LambdaRule, and then run the ELF commands in the two terminals again. You’ll see output like this in the second terminal:

INFO - Received message: {"feedback": "thing_0 said hello to Lambda!"} from topic: output/thing_0
INFO - Received message: {"feedback": "thing_1 said hello to Lambda!"} from topic: output/thing_1

What’s Next?

You now have two templates that you can use as a starting point to develop rules that republish messages. You created a rule using the SQL rules engine in AWS IoT that invokes Lambda functions written inline in Python. As you build new templates with new rules, you can use AWS CloudFormation to make sure they’re set up consistently, repeatably, and easily. What will you connect AWS IoT to next?

The Internet of Things on AWS – Official Blog

Bites of IoT: Creating AWS IoT Rules with AWS CloudFormation

Configure the CLI

What Are Rules?

Creating a SQL-Only Rule with AWS CloudFormation

YAML AWS CloudFormation Template for SQL-Only Rule

YAML AWS CloudFormation Template Explained

SQLRepublishTopicRule Section

SQLRepublishRole Section

Deploying the SQL-Only Rule

Testing the SQL-Only Rule

Creating a Rule with AWS CloudFormation to Route Messages to AWS Lambda

YAML AWS CloudFormation Template for AWS Lambda Rule

YAML AWS CloudFormation Template Explained

LambdaRepublishRole Section

LambdaHelloFunction Section

LambdaInvocationPermission Section

Deploying and Testing the AWS Lambda Rule

What’s Next?

Resources

Follow