Route data directly from IoT Core to your web services

In this blog post, we will explain how you can use AWS IoT Core Rules Engine’s HTTP action to send data directly to your existing HTTPS endpoint without writing additional code or making changes to published code. With this feature, you can integrate AWS IoT with your own web services without additional latency and complexity associated with the use of an intermediate service. For more information about this feature, refer to the HTTP action documentation. If you are new to AWS IoT rules, check out this Tutorial.

This blog assumes that:

You have access to AWS IoT console or have AWS CLI installed;
You have an existing HTTPS endpoint with a valid certificate signed by a public certificate authority (CA); and,
You can access that HTTPS endpoint’s logs or can read messages sent to that endpoint.

We will walk you through the following:

Create a topic rule with an HTTP action using AWS console;
Confirm and enable a topic rule destination using AWS console;
Create a topic rule with an HTTP action using AWS CLI;
- Define an HTTP action endpoint with and without substitution templates,
- Define an HTTP action with headers and authentications,
Manage topic rule destinations using AWS CLI;
- Create topic rule destinations,
- Confirm topic rule destinations,
- Update topic rule destinations,
- List topic rule destinations,
- Delete topic rule destinations.

Ready to get started?

In order for your web service to receive data from AWS IoT, you first need to create a topic rule with a new HTTP action. HTTP action will POST message payloads as well as any headers specified in the action to the designated HTTPS endpoint every time the action is executed. Because you can ingest data from your endpoint using HTTPS POST, you will not need to make any code changes for your endpoint to integrate with AWS IoT.

Create a topic rule with an HTTP action via AWS Console

The steps below detail how to create a topic rule with HTTP action via the AWS IoT console:

1. In the AWS IoT console, in the navigation pane, choose Act, then Rules.

2. On the Rules page, choose Create.

3. On the Create a rule page, enter a name and description for your rule.

4. Under Rule query statement, choose the latest version from the Using SQL version list. For rule query statement, enter: SELECT * FROM ‘my/topic’

This will take the entire payload data that was published on my/topic and execute rule actions to send it to downstream services. If you are new to writing AWS IoT Rules SQL statement, please see the SQL Reference page.

5. Choose Add Action.

6. On Select an action page, choose Send a message to a downstream HTTPS endpoint.

This is where you can specify the URL of your HTTPS endpoint. When this action executes, Rules Engine will construct a POST request to that endpoint, add headers specified in the action, and add the message payload to the request body after SQL statement evaluation.

7. On Configure action page, enter HTTPS Endpoint to point to your existing HTTPS endpoint, e.g. https://www.example.com/ingest.

HTTPS Endpoint supports substitution templates. As a result, there are many ways you can specify an endpoint. See Define HTTP Action Endpoint section below to learn more about ways to configure an HTTPS endpoint.

8. Choose Add Action.

9. Choose Create rule.

At this point, you have a topic rule that would take data published on AWS IoT’s ‘my/topic’ and send it to https://www.example.com/ingest. It is very important to note that in order for this to flow through properly (and not be blocked by Rules Engine), you must first validate that you control the endpoint.

Introducing the concept of Topic Rule Destinations

To ensure data is sent to an endpoint that you control, AWS IoT Rules Engine requires a confirmation that you can access the data sent to that endpoint. To keep track of your endpoints, a new AWS IoT Topic Rule Destination resource is added. Rules Engine will try to create a new topic rule destination when you create or update a topic rule with HTTP action.

When a topic rule destination is created for HTTP action, Rules Engine issues a challenge message to the specified endpoint with a token. If you can get the token and send that back to Rules Engine, then you can prove that you have access to the data on the receiving side, which would complete the confirmation process. Endpoint confirmation is a one-time operation. Once confirmed and enabled, the same endpoint can be used for multiple rules without additional confirmations.

Note that the token is valid for three days, after which you need to create a new HTTP action or follow the steps in Update Topic Rule Destination section to restart the confirmation process. See the Working with HTTP Action Destinations documentation for more details.

Confirm and Enable Topic Rule Destinations via AWS Console

The steps below show you how to confirm and enable a destination via the AWS IoT console:

1. In the AWS IoT console, in the navigation pane, choose Act, then Destinations.

2. Select a Destination entry that has your endpoint, e.g. https://www.example.com/ingest

3. In the details of that Destination, select Actions then choose Confirm and Enable.

You should see a dialog like this for your HTTPS endpoint.

A confirmation token is required to proceed. When a destination is created, Rules Engine issues a challenge message to your endpoint. That message contains the confirmation token and has a format like this:

HTTP POST {confirmationUrl}/?confirmationToken={confirmationToken}

Headers:
    x-amz-rules-engine-message-type: DestinationConfirmation
    x-amz-rules-engine-destination-arn: <Destination ARN>
    Content-Type: application/json
    ...

Body: 
{ 
    "arn": <ARN for the destination that needs to be confirmed>,
    "confirmationToken": <Confirmation Token, same as {confirmationToken} in URL>,
    "enableUrl": <Callback URL to complete the confirmation. Make HTTP GET request to this URL to finish confirmation>,
    "messageType": "DestinationConfirmation"
}

If you control that endpoint, you can simply look in your service logs for POST requests to that resource with the confirmationToken URI query, extract that token value, and paste it back into the AWS IoT console.

Tip: Make sure to enable URI Query (cs-uri-query) W3C Logging Field for this endpoint in your server, otherwise query parameters with confirmationToken could get filtered out.

4. Enter {confirmationToken} in Confirm Destination dialog.

5. Choose Confirm and Enable.

At this point, if the token is correct, the destination entry for your endpoint should show as Enabled. AWS IoT Rules Engine will now send data to that endpoint any time your rule is triggered.

PLEASE NOTE THAT AT THE POINT OF WRITING THIS BLOG:

HTTP action only supports POST to HTTPS endpoint with a valid certificate signed by a public certificate authority (CA).
HTTP action only supports ports 443 and 8443.
HTTP action has a timeout limit of 3 seconds to complete HTTP request(s) (including retries).
Standard EC2 Data Transfer costs listed here apply to HTTP action.

For the latest limits on HTTP action, please see the AWS IoT Limit page.

Create a topic rule with an HTTP action via AWS CLI

You can also create topic rules with HTTP action, as well as manage topic rule destinations using AWS CLI. You can use the following command line to create a topic rule.

Create A Topic Rule

aws iot create-topic-rule --rule-name test_http_action --topic-rule-payload file://<path/to/your/file.json> --region <your_region>

topic-rule-payload describes the topic rule as well as action(s) associated with it. You can find examples of topic-rule-payload below.

When creating topic-rule-payload, you will need to specify url, an HTTP endpoint where your web service can receive messages. url supports substitution templates. As a result, there are many ways you can configure url, and each has its own rule to follow (no pun intended). We created five examples below to explain how you can define this endpoint in your topic-rule-payload.

Define HTTP Action Endpoint

Option 1: Hard-coded endpoint

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

In this example, url is hard-coded, meaning no substitution template is used. Once this action is added, Rules Engine automatically creates a topic rule destination for https://www.example.com URL and issues a challenge to that URL. Once you confirm and enable the URL by responding to the challenge, Rules Engine will deliver data to that endpoint. A confirmationUrl is not required in this case since it can be inferred from url value. This is only possible if url has no substitution templates.

Option 2: Hard-coded endpoint with sub-paths or query parameters

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com/subpath?myqueryparams=staticdata",
                "confirmationUrl": "https://www.example.com"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

This option is similar to Option 1, but you can also specify a confirmation (base) URL. confirmationUrl is used to create the topic rule destination, which is the destination for the confirmation challenge. It is required to be a prefix of the url where message payload is delivered. Once you confirm and enable confirmationUrl, the base URL (https://www.example.com) and all sub paths/query parameters will be allowed for all rules.

Option 3: Dynamic endpoint with hard-coded domain name

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com/${clientid()}/data",
                "confirmationUrl": "https://www.example.com"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

For this option, url is not determined until a message is processed, therefore, confirmationUrl is required. Once confirmationUrl is enabled, Rules Engine can send data not only to https://www.example.com, but to any sub paths/query parameters for this destination. If the destination was not previously enabled, creating this action will automatically create the topic rule destination based on confirmationUrl. You will need to enable that destination before Rules Engine can send messages to url.

Option 4: Dynamic endpoint with dynamic domain name

{
    "actions": [
        {
            "http": {
                "url": "https://${value_from_payload}.example.com/${clientid()}/data",
                "confirmationUrl": "https://${value_from_payload}.example.com"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

This is the most complicated scenario, because both url and confirmationUrl are not known until a message is evaluated. In such cases, Rules Engine will not automatically create a topic rule destination when the rule is created. Rules Engine has two requirements:

confirmationUrl with substitution templates (${value_from_payload}) needs to be a prefix of url; and,
Customer needs to manually create, confirm and enable all possible topic rule destinations for confirmationUrl after substitutions.

In the example above, confirmationUrl is the prefix of url, so the rule gets successfully created. Suppose ${value_from_payload} is a set of possible values (“test”, “prod”, “beta”). You would need to manually create and enable topic rule destinations for these possible confirmationUrl values: (https://test.example.com, https://prod.example.com, https://beta.example.com). See Manually Create Topic Rule Destination section below to learn how to create topic rule destinations using AWS CLI.

When a message gets evaluated, Rules Engine checks if confirmationUrl after substitutions was confirmed and is Enabled. So, if ${value_from_payload} gets substituted to “beta”, Rules Engine checks if https://beta.example.com destination has been confirmed and is Enabled before sending data to https://beta.example.com/123456789012/data.

Option 5: Fully dynamic endpoint

{
    "actions": [
        {
            "http": {
                "url": "${full_control_from_payload}",
                "confirmationUrl": "${full_control_from_payload}"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

This option is very similar to Option 4. The value of confirmationUrl must be the prefix of url, and you need to manually create, confirm, and enable all possible destinations for confirmationUrl after substitutions. In addition, Rules Engine makes sure data is targeting HTTPS endpoints on port 443 (default) or 8443.

Now let’s look at how you can add headers to your HTTP action.

Define HTTP Action with Headers

Headers are defined in key-value pairs and you can specify up to 100 headers in each HTTP action. Note that only value field supports substitution templates, and certain headers are implicitly defined by default (e.g. Content-Type, User-Agent). You can overwrite Content-Type header via headers key-value pair.

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com",
                "headers": [
                    {
                        "key": "static_header_key",
                        "value": "static_header_value"
                    },
                    {
                        "key": "substitutable_header_key",
                        "value": "${value_from_payload}"
                    }
                ]
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

Define HTTP Action with Authentications

HTTP action supports authentication methods such as Basic Auth and API key based authentication. The examples below teach you how to configure these authentication methods using HTTP headers and query parameters.

Define Basic Authentication

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com",
                "headers": [
                    {
                        "key": "Authorization",
                        "value": "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="
                    }
                ]
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

Following ‘Basic’ HTTP Authentication Scheme defined by Internet Engineering Task Force (IETF), this HTTP action will construct an HTTP POST request with header Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== to https://www.example.com.

Define API Key based Authentication

{
    "actions": [
        {
            "http": {
                "url": "https://www.example.com/something?api_key=abcd1234",
                "confirmationUrl": "https://www.example.com"
            }      
        }
    ],
    "sql": "select * from 'a/b'",
    "awsIotSqlVersion": "2016-03-23",
    "ruleDisabled": false
}

In this example, the endpoint authenticates using an API key query parameter. Alternative query parameters or headers can be used to specify the API key, depending on the target service implementation.

Manage Topic Rule Destinations using AWS CLI

In this section we will show you how to create, confirm, update, list, and delete topic rule destinations using AWS CLI.

Manually Create Topic Rule Destination

aws iot create-topic-rule-destination --destination-configuration '{"httpUrlConfiguration": {"confirmationUrl": "https://www.example.com"}}' --region <your_region>

This command will create a new topic rule destination and will issue a confirmation challenge to https://www.example.com. The topic rule destination status will be IN_PROGRESS with a status reason Awaiting confirmation. If the confirmation is not complete within three days, the state will be moved to ERROR.

You will need to use this command line to manually create topic rule destinations if confirmationUrl of your HTTP action contains substitution templates.

Confirm Topic Rule Destination

aws iot confirm-topic-rule-destination --confirmation-token 'confirmationToken' --region <your_region>

Once you have the confirmation token from the confirmation message, you can use the command line above to confirm the corresponding topic rule destination.

For security reasons, the topic rule destination will be in DISABLED status after confirmation. You will need to enable the destination using the update-topic-rule-destination command line (see below) or in the console in order for Rules Engine to send traffic to this destination.

Update Topic Rule Destination

aws iot update-topic-rule-destination --arn 'arn' --status 'ENABLED|DISABLED|IN_PROGRESS' --region <your_region>

You can use this command line to enable/disable a topic rule destination, or resend a new confirmation message to the destination. An arn is used to identify the topic rule destination, and you can receive the arn via the list-topic-rule-destination command line (see below) or in the console. status is the desired new status state for a destination. Note that status must be in the ENABLED state in order for Rules Engine to send traffic to this destination.

To resend the confirmation challenge message, you can update the topic rule destination status to IN_PROGRESS. For more information see the UpdateTopicRuleDestination API documentation.

List Topic Rule Destination

aws iot list-topic-rule-destinations --max-results <value> --region <your_region>

The command line above lists your topic rule destinations including arn of each destination.

Delete Topic Rule Destination

aws iot delete-topic-rule-destination --arn 'arn' --region <your_region>

To delete a topic rule destination, simply use the command line above with the arn of the destination.

Ready to send data from AWS IoT Core directly to your own web services for processing without writing a single line of code? HTTP action is available in all AWS regions where AWS IoT Core is available, and with this addition, AWS IoT Core now supports 17 action types. For more information, refer to the AWS IoT documentation.

The Internet of Things on AWS – Official Blog