The Internet of Things on AWS – Official Blog

5 tips to build AWS IoT Greengrass v2 Components

AWS IoT Greengrass v2, announced on December 2020, is an open source edge runtime service that you can use to build, deploy and manage edge software components and locally act on the data that your intelligent IoT devices capture. For example, you can run data prediction machine learning models, filter and aggregate device data as you desire, and use pre-built software components, features, and tools to effectively manage your device fleets at scale.

With AWS IoT Greengrass components, which consist of application and runtime libraries, you can develop custom applications, test and deploy them on your AWS IoT Greengrass core device – giving you more flexibility over AWS IoT Greengrass v1. Additionally, components can run outside of containers (new in 2.0), which allows you to work directly with local resources on your core devices. Also, in AWS IoT Greengrass v1, AWS Lambda functions define the software that runs on core devices while in AWS IoT Greengrass v2, components can consist of any software application you define.

Every component is composed of a recipe and artifacts. The recipe file defines the component‘s metadata. This includes the component’s configuration parameters, component dependencies, lifecycle, and platform compatibility. The lifecycle defines the commands to install, run, and shut down the component. The recipe can be defined in YAML and JSON format. Artifacts are optional and consist of component binaries and may include scripts, compiled code, static resources, and any other files that a component consumes.

In this post, we are sharing 5 tips for you to consider while developing AWS IoT Greengrass v2 components. These tips and insights may help you define mechanisms for structuring AWS IoT Greengrass components. Also, this process helps you accelerate and improve your development workflow and get started more quickly with the component development.

The prerequisites for following this post are:

  1. You have a fully functional AWS user account. For set up instructions, refer to our documentation.
  2. You have a machine with AWS IoT Greengrass installed and fully operational. If you haven’t done this yet, follow the installation instructions.

Let’s get started.

1) Use Command Line Interface (CLI) commands

During component development, there are a few CLI commands that give you quick and easy insights into the current status of AWS IoT Greengrass Core software and your components. All the commands in this blog post are for Linux or Unix based systems. If you use a different operating system, adjust those accordingly.

By default, the AWS IoT Greengrass Core software writes logs to only the local file system. You can view file system logs in real time to debug your AWS IoT Greengrass components. There are two log types that are especially important:

  1. The AWS IoT Greengrass Core software log file. It contains real-time information about components and deployments. This is the first place to look when you start a deployment and want to know what is going on. The errors in the log file then might help you troubleshoot. Since the directory of the log-files is owned by root, we use sudo to access those files. 
    $ sudo tail -F /greengrass/v2/logs/greengrass.log

    2.     AWS IoT Greengrass component log files. These files contain real-time information about the corresponding component that runs on the device. Generic and Lambda components write standard stdout and stderr to these log files, with your own components you can use these functionalities according to your needs.

    $ sudo tail -F /greengrass/v2/logs/<component-name>.log
# Concrete example: Your component is named ggv2.custom-comp.logging
$ sudo tail -F /greengrass/v2/logs/ggv2.custom-comp.logging.log

To get better insights into components and their status, there are some very useful AWS IoT Greengrass CLI commands to list and restart components. Before you can use the commands, you need to install the AWS IoT Greengrass CLI first (follow the instructions later in this Blog to do so). The list command gives you an output of component’s name, its version, and its current state.

$ sudo /greengrass/v2/bin/greengrass-cli component list
Components currently running in Greengrass:
Component Name: FleetStatusService
Version: 0.0.0
State: RUNNING
Configuration: {"periodicUpdateIntervalSec":86400.0}
Component Name: UpdateSystemPolicyService
Version: 0.0.0
State: RUNNING
Configuration: null
Component Name: aws.greengrass.Nucleus
Version: 2.0.0
State: FINISHED
Configuration: {"awsRegion":"region","runWithDefault":{"posixUser":"ggc_user:ggc_group"},"telemetry":{}}
Component Name: DeploymentService
Version: 0.0.0
State: RUNNING
Configuration: null
Component Name: TelemetryAgent
Version: 0.0.0
State: RUNNING
Configuration: null
Component Name: aws.greengrass.Cli
Version: 2.0.0
State: RUNNING
Configuration: {"AuthorizedPosixGroups":"ggc_user"}

When you restart a component, the core device uses the latest changes.

$ sudo /greengrass/v2/bin/greengrass-cli component restart \
--names "<component-name>"
# Concrete example: Your component is named ggv2.custom-comp.logging-1.0.1
$ sudo /greengrass/v2/bin/greengrass-cli component restart \
--names "ggv2.custom-comp.logging-1.0.1"

You can get more insights for monitoring AWS IoT Greengrass logs, how to enable logging those to Amazon CloudWatch, as well as useful AWS IoT Greengrass CLI commands.

2) Develop components locally

For a great experience while developing components, we recommend doing so locally and then creating the component in the cloud to deploy to your AWS IoT Greengrass Core devices. Let’s go through a simplified development process for our example component ggv2.custom-comp.logging. We can use this developed component later on to publish to the cloud and deploy to core devices.

A/ Folder structure

Lets look at the sample folder structure used thoughout this blog post for our sample component ggv2.custom-comp.logging:

~                                           <-- Your environment
├── GreengrassDev/
│   ├── deployment.json
│   └── components
│       └── ggv2.custom-comp.logging
│           └── recipes
│              └── ggv2.custom-comp.logging.yaml
│           └── artifacts
│              └── ggv2.custom-comp.logging
│                 └── 1.0.0
│                    └── src
│                       └── script.py
│                    └── main.py
├── GreengrassCore                          <-- GreengrassCore installation
└── README.md

Next to the GreengrassCore directory, which gets created during installation of the AWS IoT Greengrass Core software, we create a GreengrassDev directory with the subdirectory components, which contains the folders recipes and artifacts. For our artifacts, it is important that the structure adheres to artifacts/<componentName>/<componentVersion>.

B/ First deployment including AWS IoT Greengrass CLI

To make use of the AWS IoT Greengrass CLI, we have to create a deployment.json file with the following content:

{
   "targetArn": "arn:aws:iot:<region>:<account-id>:thing/<thing-name>",
   "deploymentName": "gg-iot-ggv2-bp-deployment",
   "components": {
      "aws.greengrass.Nucleus": {
         "componentVersion": "2.9.3"
      },
      "aws.greengrass.Cli": {
         "componentVersion": "2.9.3"
      }
   }
}

Replace the placeholders for your region, account, and thing-name with your actual values. In the following examples, we assume the region is eu-central-1, the account-id is 123456789 and the name of the core device is aws-iot-ggv2-bp-core. You also have the possibility to target groups of core devices in your deployment, which makes it easy to keep software on multiple devices up to date, which is one of the features launched with AWS IoT Greengrass v2 to improve the customer experience. So in the deployment.json file, the targetArn can point to the Amazon Resource Name (ARN) of a thing or a group.

To create a deployment to the cloud, we use the AWS CLI:

$ aws greengrassv2 create-deployment --cli-input-json file://deployment.json

Check in the Console (or Logs) to see if the deployment was successful.

C/ Development helpers: environment variables and functions

Let’s define all the important aspects around our component, its version, and folder structure:

$ export COMPONENT_NAME="ggv2.custom-comp.logging"
$ export COMPONENT_VERSION="1.0.0"
$ export RECIPE_DIR="~/GreengrassDev/components/${COMPONENT_NAME}/recipes"
$ export ARTIFACT_DIR="~/GreengrassDev/components/${COMPONENT_NAME}/artifacts"

To simplify our local development process, we create two helper functions. Helper function gg_deploy takes the exported variables and creates a deployment for our specified COMPONENT_VERSION. The second helper function gg_delete, deletes any previously deployed component-version. Instead of updating the versions with every new deployment, we delete the previously deployed component-version with this function, which allows us to deploy the new version using the same specified COMPONENT_VERSION thereafter. In both functions we use the AWS IoT Greengrass CLI deployment create commands with different parameters.

#Deploy-Helper-Function
$ gg_deploy(){
sudo /greengrass/v2/bin/greengrass-cli deployment create \
--recipeDir $RECIPE_DIR --artifactDir $ARTIFACT_DIR \
--merge "$COMPONENT_NAME=$COMPONENT_VERSION";
}

#Remove-Helper-Function
$ gg_delete(){
sudo /greengrass/v2/bin/greengrass-cli --ggcRootPath /greengrass/v2 deployment create \
--remove "$COMPONENT_NAME";
}

Let’s have a look into the files in our artifacts (~/GreengrassDev/components/artifacts/ggv2.custom-comp.logging/1.0.0) folder, which are our main.py and the script.py, which is placed in our src folder. The code in script.py looks like this (this is a simplified example):

import datetime
import time

def loopfunction():
    while True:
        message = f"Hello! Current time: {str(datetime.datetime.now())}."
    
        # Print the message to stdout.
        print(message)
        
        time.sleep(1)

As well as our main function in main.py:

import sys
import src.script as helloworld

def main():
    helloworld.loopfunction()

if __name__ == "__main__":
    main()

Also, the content of your recipe file ggv2.custom-comp.logging.yaml is:

---
RecipeFormatVersion: "2020-01-25"
ComponentName: "{COMPONENT_NAME}"
ComponentVersion: "{COMPONENT_VERSION}"
ComponentDescription: "Sample Component"
ComponentPublisher: "Me"
Manifests:
  - Platform:
      os: linux
    Lifecycle:
      Run: "python3 -u {artifacts:path}/main.py"

An example process for our component, with two changes that happen incrementally, could look like this:

# Substitute the right component name and version into the recipe file
$ sed -i 's/{COMPONENT_NAME}/'"$COMPONENT_NAME"'/g' $RECIPE_DIR/$COMPONENT_NAME.yaml
$ sed -i 's/{COMPONENT_VERSION}/'"$COMPONENT_VERSION"'/g' $RECIPE_DIR/$COMPONENT_NAME.yaml
# Deploy-Helper-Function
$ gg_deploy
# Do changes to the script.py
$ gg_delete
$ gg_deploy
# Do changes to the script.py and the recipe file
$ gg_delete
$ gg_deploy
# Once we are done and happy with our developed version, we can deploy using the cloud
# I recommend using a last gg_delete to avoid a mismatch in versions
$ gg_delete

This concludes the segment on local development. We create the component in the cloud and deploy it to the core devices.

3) Create the component in the cloud and deploy to core devices using the AWS IoT Greengrass Development Kit (GDK)

To ease the development process for custom components, AWS published the AWS IoT Greengrass Development Kit (GDK) CLI, which is available open source under the Apache-2.0 license. It helps you create, build, and publish custom components. Refer to the links for the prerequisites when using the AWS GDK CLI, as well as the installation process.

Let’s look at the development process here. To get started, either use a template or a community component. In our case, we use helper functions and environment variables to use our previously created local component as basis for building and publishing the component with the GDK. The AWS IoT GDK CLI updates the version and artifact URIs for you each time you publish a new version of the component.

#Prepare-GDK-Helper-Function
$ gg_gdk_prepare(){
mkdir -p $RECIPE_DIR/../greengrass-gdk-prepare/$COMPONENT_NAME && cd "$_"
cp -a $ARTIFACT_DIR/$COMPONENT_NAME/$COMPONENT_VERSION/. .

cp -a $RECIPE_DIR/$COMPONENT_NAME.yaml ./recipe.yaml
sed -i 's/'"$COMPONENT_VERSION"'/{COMPONENT_VERSION}/g' recipe.yaml
sed -i 's/{artifacts:path}/{artifacts:decompressedPath}\/'"$COMPONENT_NAME"'/g' recipe.yaml

touch gdk-config.json
}
# Execute the function
$ gg_gdk_prepare

Before creating the component, we now need to update the recipe file to point to the Amazon S3 bucket by adding the Artifacts section to the Manifest-part of the recipe.yaml file:

---
RecipeFormatVersion: "2020-01-25"
ComponentName: "ggv2.custom-comp.logging"
ComponentVersion: "{COMPONENT_VERSION}"
ComponentDescription: "Sample Component"
ComponentPublisher: "Me"
Manifests:
  - Platform:
      os: linux
    Artifacts:
      - URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/ggv2.custom-comp.logging.zip"
        Unarchive: ZIP
    Lifecycle:
      Run: "python3 -u {artifacts:decompressedPath}/ggv2.custom-comp.logging/main.py"

Finally, let’s copy the following code into the  gdk-config.json file. Replace all the PLACEHOLDER with the actual values:

{
  "component": {
    "<PLACEHOLDER_COMPONENT_NAME>": {
      "author": "Me",
      "version": "NEXT_PATCH",
      "build": {
        "build_system": "zip"
      },
      "publish": {
        "bucket": "<PLACEHOLDER_BUCKET>",
        "region": "<PLACEHOLDER_REGION>"
      }
    }
  },
  "gdk_version": "1.0.0"
}

Alternatively, export the region and bucket to environment variables and then use the gg_gdk_substitute function:

#Existing S3 bucket that GDK uploads the artifacts to
$ export BUCKET_NAME="ggv2-blogpost-XXXXXXXXXX"
$ export REGION="eu-central-1"

#Prepare-GDK-Helper-Function
$ gg_gdk_substitute(){
sed -i 's/<PLACEHOLDER_COMPONENT_NAME>/'"$COMPONENT_NAME"'/g' gdk-config.json
sed -i 's/<PLACEHOLDER_BUCKET>/'"$BUCKET_NAME"'/g' gdk-config.json
sed -i 's/<PLACEHOLDER_REGION>/'"$REGION"'/g' gdk-config.json
}

#Run the function
$ gg_gdk_substitute

Next step is to run the build (gdk component build) and publish (gdk component publish) commands, which results in the component being published as custom component to your own AWS account. For more detailed instructions and explanations, visit the documentation page. Last step is to deploy the component to our core, for which we use the deployment.json file and the AWS CLI.

{
    "targetArn": "arn:aws:iot:eu-central-1:123456789:thing/aws-iot-ggv2-bp-core",
    "deploymentName": "gg-iot-ggv2-bp-deployment",
    "components": {
        "aws.greengrass.Nucleus": {
            "componentVersion": "2.9.3"
        },
        "aws.greengrass.Cli": {
            "componentVersion": "2.9.3"
        },
        "ggv2.custom-comp.logging": {
            "componentVersion": "1.0.0"
        }
    }
}

$ aws greengrassv2 create-deployment --cli-input-json file://~/GreengrassDev/deployment.json

Feel free to monitor the status of your deployment by trailing the log files and listing all running components.

If you get an PackageDownloadException with the reason Failed to head artifact object from S3, you need to adjust the Permission of your Role Alias (The Role Alias needs read-permissions for your S3 bucket that stores the artifact-files). For more details refer to Section 4 of this blog post.

4) Understand permissions within AWS IoT Greengrass v2

There are three important aspects regarding permissions that you need to be aware of:

  1. The AWS IoT Role Alias: Used when the AWS IoT Greengrass Core and deployed components interact with services outside of AWS IoT.
  2. Interprocess Communication (IPC) Authorisation for components when they want to interact with the nucleus and other AWS IoT Greengrass components.
  3. The Thing and attached Certificate of the AWS IoT Greengrass Core device: Used to specify what the AWS IoT Greengrass Core Thing is allowed to do within AWS IoT. This also includes the permission to assume the AWS IoT Role Alias. To read up on AWS IoT Core policies, visit our documentation.

Since the first two concepts are specific for AWS IoT Greengrass v2, we go deeper in the next two subchapters.

A/ AWS IoT Role Alias

Devices use their X.509 certificates to get temporary AWS credentials by calling the credential provider services, which reduces the need to store AWS access key ID and secret access key ID on the AWS IoT Greengrass core device. The AWS IoT role alias within AWS IoT Core points to the AWS Identity and Access Management (IAM) role that allows AWS IoT Greengrass to communicate with services outside of AWS IoT (Default role alias: GreengrassV2TokenExchangeRoleAlias). For more information, see Authorizing direct calls to AWS services in the AWS IoT Core Developer Guide.

The first point where you normally need permissions is when you upload your artifacts to S3 and want to deploy one of your components in the cloud to your device. In order for your device to download the S3 artifacts, the IAM role needs to allow s3:GetObject permissions for the bucket your artifacts got uploaded to. For more detailed information on permissions as well the default allowed operations, check out the documentation on the device service role. If docker is used, you need to additionally grant access to Amazon Elastic Container Registry (ECR).

B/ Interprocess communication permissions

If you want to allow your components to interact with other components, the Greengrass nucleus, or AWS IoT Core, then they need to use Interprocess communication. To do so, the permission for that communication needs to happen on component level within the recipe file.

The accessControl section within the recipe may then be extended by blocks concerning the different IPC service identifier:

{
  "accessControl": {
    "<IPC service identifier>": {
      "<unique identifier>": {
        "policyDescription": "Allows access to [...]",
        "operations": [
          "<operation 1>",
          "<operation 2>"
        ],
        "resources": [
          "*"
        ]
      }
    }
  }
}

For the different IPC service identifier as well as operations, refer to the documentation. Lets have a look at our component ggv2.custom-comp.logging. Let’s assume that the component wants to publish to an AWS IoT Core cloud topic named dt/sensorA/temperature as well as subscribe to the local topic dt/sensorB/buttonvalue. Then our updated recipe file from before would look like:

---
RecipeFormatVersion: "2020-01-25"
ComponentName: "ggv2.custom-comp.logging"
ComponentVersion: "1.0.0"
ComponentDescription: "Sample Component"
ComponentPublisher: "Me"
ComponentConfiguration:
  DefaultConfiguration:
    accessControl:
      aws.greengrass.ipc.pubsub:
        'ggv2.custom-comp.logging:pubsub:1':
          policyDescription: Allows access to subscribe to local topic dt/sensorB/buttonvalue.
          operations:
            - 'aws.greengrass#SubscribeToTopic'
          resources:
            - 'dt/sensorB/buttonvalue'
      aws.greengrass.ipc.mqttproxy:
        'ggv2.custom-comp.logging:mqttproxy:1':
          policyDescription: Allows access to publish to local topic dt/sensorB/buttonvalue.
          operations:
            - 'aws.greengrass#PublishToIoTCore
          resources:
            - 'dt/sensorA/temperature'
Manifests:
  - Platform:
      os: linux
    Artifacts:
      - URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/ggv2.custom-comp.logging.zip"
        Unarchive: ZIP
    Lifecycle:
      Run: "python3 -u {artifacts:decompressedPath}/ggv2.custom-comp.logging/main.py"

You can use wildcards in your accessControl section. Within a specified IPC service identifier, using a *-wildcard in the operation section means that all operations within that IPC service identifier are allows. Same goes for resources: using a wildcard in that section allows the specified operations for all topics. For more detailed information for each IPC service, refer to the documentation.

5) Understand the process of updating the component configuration

Let’s have a look at how component configuration updates work. The general configuration updates merge the old configuration with the new ones, which can mean one of the following:

  1. You are adding a new key with a new value. This key and value is then added to the existing configuration with the existing fields.
  2. You have an existing key, which has a string or number value. If you change that value, the new configuration contains that old key with its new value.
  3. You have an existing key, which contains a JSON subelement, where you specify one new value. Then the old key points to the old JSON, which has a new added field, which is your new value.
  4. To remove a key from the configuration, you need to use the reset-functionality.

Lets take a simplified example for merging components:

Existing Configuration Configuration Update Effective Configuration 
{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
},
"key2":"value2"
}

{
"key1":{
"subkey_1b":"new_value1b"
}
}

{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"new_value1b"
},
"key2":"value2"
}

Lets look at another example:

Existing Configuration Configuration Update Effective Configuration 
{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
},
"key2":"value2"
}

{
"key1":"new_string1"
}

{
"key1":"new_string1",
"key2":"value2"
}

If you want to delete a component configuration or a subfield, you need to use the reset functionality. You also need to use a JSON pointer to address the fields that you want to reset to default value or delete.

Existing Configuration Configuration Update Effective Configuration 
{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
},
"key2":"value2"
}

"configurationUpdate": {
"reset": ["/key2"]
}

{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
}
}

Let’s put this all together in the next example, that combines merge and reset in one configurationUpdate:

Existing Configuration Configuration Update Effective Configuration 
{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
},
"key2":"value2"
}

"configurationUpdate": {
"reset": ["/key1/subkey_1a"],
"merge": "{\"key3\":\"value3\"}
}

{
"key1":{
"subkey_1b":"value1b"
},
"key2":"value2",
"key3":"value3"
}

If we want to do a full reset of the entire configuration, we need to specify a single empty string as the reset update:

Existing Configuration Configuration Update Effective Configuration 
{
"key1":{
"subkey_1a":"value1a",
"subkey_1b":"value1b"
},
"key2":"value2"
}

"configurationUpdate": {
"reset": [""]
}

{}

If you want to know more about this, refer to the developer guide on merge updates.

Clean up

To uninstall the AWS IoT Greengrass Core software and delete the core device from the AWS IoT Greengrass service, follow the instructions in the documentation.

Conclusion

In this blog post, we went over five tips to develop AWS IoT Greengrass v2 components. These tips and insights can help you develop mechanisms for structuring AWS IoT Greengrass components and accelerate and improve your development workflow.

Those tips include some helpful CLI commands, how to develop components first locally and then use the AWS GDK CLI for publishing those. We also discussed permission management and the process of updating the component configuration.

To learn more about AWS IoT Greengrass v2, check out the Documentation as well as the Developer Guide.