中文版 – Twelve months ago, we started working on AWS RoboMaker, a cloud robotics service. One of the biggest questions looming over us was: How are we going to make it easy to run any ROS application on our service? Robot applications are a large mix of different packages with numerous dependencies. When simulation is mixed in, that list of dependencies grows. After much thought and research, we were inspired by appimage, flatpak, and snapcraft to create a single file format that can run in a local development environment and on our service. We call this format a bundle.

After deciding what kind of packaging artifact to design, we wanted to create a command line tool to make it easy to generate. It was very important to us that the tool fit within the existing ROS ecosystem. We decided to build on top of colcon, the latest and greatest in ROS build tooling. Colcon can build ROS1 and ROS2 applications. It is also highly extensible, and provides many of the important features we needed, out of the box.

This article explains the existing ROS build tooling and why colcon was chosen as the build tool for ROS2. Colcon does a ton of the heavy lifting for us, allowing us to focus on our specific functionality. In addition, colcon build replaces catkin_make and ament_make with no modifications to the packages being built. This means that, by building our tooling on top of colcon, AWS RoboMaker users can install a single tool and execute the full workflow to generate a bundle for use with AWS RoboMaker.

In this post, I will go over the normal workflow for creating a bundle to use with AWS RoboMaker, and some solutions for frequently-encountered issues. In the second section, I will dive deeper into the newest version of our ‘bundle’ file format and provide an overview of what happens when you execute colcon bundle to create a bundle.

Why a Bundle?

ROS simulation workflows require an environment with ROS and other runtime dependencies of the application installed. This is usually fulfilled by executing various apt-get install commands. That install takes a long time, and cannot be reproduced reliably because upstream package updates can occur at any time. For our service, we were concerned about the reliability of using apt to install a new environment for every simulation run, so we looked into various complete packaging formats. The most compelling and reliable distribution format for an entire ROS application was a Debian package. It can contain everything required to run the robot: ROS, boost or other external libraries, and the actual application code.

We decided not to go with a preexisting format because we wanted flexibility to add optimizations for downloading, creation, and updates. So we created the ‘bundle’ format and the corresponding tooling to support it. Our first iteration worked well, but it had some limitations:

We are excited to announce the second version of the bundle format with optimizations to reduce or remove these limitations! The format now supports partial downloads and partial extraction. This has three major benefits:

AWS RoboMaker supports this new version transparently. If you would like to try out these new improvements run sudo pip3 install -U colcon-bundle in your development environment. I will dive deeper into the implementation of this new format in the second part of this blog. If you have any questions, ideas, or issues, we’d love to read them in our issues on GitHub.

Using colcon to Build a Bundle for AWS RoboMaker

Bundling your workspace with colcon takes only two steps:

Build

Every ROS package that can be built with catkin_make can also be built with colcon build. Over months working with ROS packages, we’ve encountered only minor issues building packages with colcon build. If you run into issues building packages with colcon build that build successfully with catkin_make, please report them in colcon-ros issues.

NOTE: The most noticeable change from catkin_make to colcon build is that there is no longer adevel directory. If you want to add your workspace to your local environment, execute source install/setup.sh instead.

Bundle

After your workspace has been built with colcon build, you should have a build and an install directory. To start bundling, execute colcon bundle. Our tool inspects your dependencies, downloads all of them, and compresses them along with your built workspace into a bundle to be used on AWS RoboMaker. The built bundle is located at bundle/output.tar

Common Build Issues

The standard build instructions for ROS1 development use catkin_make. Because colcon is relatively new, many ROS1 developers might not be familiar with it, but it is fully compatible with catkin builds. While building existing ROS packages (previously built with catkin_make) with colcon build we ran into a couple of minor issues, and want surface them here to save other developers some time and effort.

CMAKElists.txt in Nested Folders

A common build issue we encountered is colcon failing to properly enumerate all the packages in the workspace, leading to failed colcon bundle invocations and odd installation directory structures. This is usually caused by nested folder structures withCMakeLists.txt in the intermediate directories. This is required for building nested packages within directories withcatkin_make, but causes trouble with colcon. Colcon supports arbitrarily deep nested folder structures, and finds your packages automatically. Here’s a diagram for reference:

src/
├── package_1/
│ ├── package.xml
| └── CMakeLists.txt (OK)
└── intermediate_directory/
    ├── package_2
    | ├── package.xml
    | └── CMakeLists.txt (OK)
    ├── package_3
    | ├── package.xml
    | └── CMakeLists.txt (OK)
    └── CMakeLists.txt (REMOVE)

Missing Install Directives in CMAKELists.txt

Another issue we’ve run into is missing files. When using catkin_make, the devel directory and itssetup.sh adds all the local packages into the search paths for ROS tools. Colcon works differently: it installs all the targets you specify into theinstall directory, which means that all your CMake install() directives are executed. If you are experiencing errors like: [a_launchfile] is neither a launch file in package [a_package] nor is [a_package] a launch file name or [rosrun] Couldn't find executable named my_node below /opt/ros/kinetic/share/my_package, then adding calls to install() in yourCMakeLists.txt could fix the issue. There are excellent examples of how to do this on the ROS Wiki.

Common Bundling Issues

These are the most frequent issues we’ve run into while working with bundles of applications on AWS RoboMaker.

Cannot Locate rosdep Definition for [Package Name]

Ensure that you are using the correct ROS package name for the package you want to depend on. A package might be named ros-kinetic-packagename in apt, but in your package.xml it should be packagename. Search the ROS distro GitHub repo to see if the package is in the existing rosdep database. If it is not, you can follow one of the tutorials for adding dependencies to rosdep.

Missing Dependencies

The most common problem we saw while bundling various open source ROS libraries was missing dependencies. Much of the time, the build environment already has a dependency installed because of another package, and everything builds fine, but after the package is bundled that dependency is missing. The common error messages when a dependency is missing are Could not load libxzy.so, No such file or directory some_script.py, or Could not load module 'python_dependency'. To solve this, add the dependency to the package.xml of the package that requires it, and re-bundle.

If you use dependencies in your applications from your own apt or pip repository, you need to include those repositories in your colcon bundle invocation. To resolve this, try the following:

Build and Bundle Complete

Now you can build and bundle your application to simulate on AWS RoboMaker! To upgrade to the latest version of our packages run: sudo pip3 install -U colcon-bundle colcon-ros-bundle in your development environment. If you have any questions or run into issues, please post on the relevant GitHub repo, we will be happy to help. If you’re interested in what is actually happening when colcon bundle is executed, read on!

Bundle Deep Dive

This section is a technical deep dive into the internals of the bundle format and the implementation of colcon bundle. At the end of this section you will come out with an understanding of what’s happening when colcon bundle executes. After you see how easy it is, we hope you’ll want to build your tooling on top of the colcon ecosystem and contribute it back to the community.

Bundle File Format

A ‘bundle’ is the packaging format used by AWS RoboMaker to run an entire ROS application. It contains the entire transitive closure of robot or corresponding simulation application. ROS has already defined a standard packaging format which specifies how to declare dependencies and what build tool to use. Using colcon, our tooling collects this information, invokes various package installers (currently pip and apt) to pull down the dependencies, installs those dependencies into a local staging area, and then packages them along with the local built workspace into a bundle for consumption by AWS RoboMaker.

That workflow produces what we call a ‘bundle’ (we know, it’s a boring name – if you have a naming idea for this format, please open an issue in our repository!). The specification of the bundle format is located in the colcon-bundle repository. I’ll give an overview here.

This diagram shows V2 of our bundle format. You can view the history here. A ‘bundle’ file follows the standard GNU tar format. It has a minimum of three components:

I outlined the benefits of the latest version of our format up above. These are the major changes between V1 and V2:

Overall, we are very happy with these improvements. If you have any ideas, we’d love to read them in our issues on GitHub.

colcon bundle Execution Flow

In this section I will dive into what is actually happening when you execute colcon bundle. (It has been a great experience building our tooling on top of colcon. If you are looking to create specialized build tooling, I highly recommend seeing if colcon could provde value as a base framework.)

The colcon ecosystem is made up of individual Python packages that rely only on direct dependencies and colcon-core as a base. Almost all of the functionality ofcolcon is modeled as ExtensionPoints defined by base infrastructure in colcon-core. colcon leverages setuptools entry points as its plugin mechanism to fulfill its goal of modularity and loose coupling.

To extend the command line interface, colcon supports adding commands via VerbExtensionPoints. To build our functionality, we created the bundle verb and registered it in our setup.cfg. That’s all we had to do to get an integrated command into this awesome tool. (All the work from there on was specific to our domain.) If you’re looking to build tooling around your ROS workflows, use colcon! I can’t recommend it enough.

Once colcon bundle has been invoked, a series of actions occur, some provided to us by thecolcon infrastructure, and some written by us. I’ll outline them here, but I encourage you to dive into the code for a deeper understanding of how everything works.

1. Our VerbExtensionPoint, BundleVerb, is invoked. We are passed the context which contains all the arguments we registered along with arguments from other extension points.
2. We then invoke some of the colcon-core functionality to crawl the workspace file tree and identify packages. This then creates a PackageDescriptor for each package to be included in the execution of bundle. Package identification is done using PackageIdentificationExtensionPoints. The extension registered to identify ROS packages is RosPackageIdentification defined in colcon-ros. When it identifies a ROS package, it sets the type in the PackageDescriptor to ros.cmake, ros.catkin, or ros.ament.
3. Once all the packages have been identified colcon executes any PackageAugmentationExtensionPoints that have been registered for a given package type. Along with identification, RosPackageIdentification also provides an augmentation that parses the package’s package.xml and adds build, test, and run-time dependencies to the package descriptor metadata.
4. Now back into our code, we inspect the dependency metadata attached to each PackageDescriptor to determine if anything has changed since the last invocation. If nothing has changed, we skip further dependency analysis and continue to bundle up the built workspace with the existing dependency overlay. If we determine that dependencies have changed, we continue with full dependency analysis.
5. During full dependency analysis, for each PackageDescriptor, we invoke its corresponding bundle task. We currently support ros.catkin, ros.cmake, and python package types. This support is registered in colcon-bundle/setup.cfgand colcon-ros-bundle/setup.cfg. New package types can be supported by adding a corresponding bundle task. For each package type, the corresponding bundle task collects the dependencies and then invokes the appropriate BundleInstallerExtensionPoint for each dependency. Our ROS tasks use rosdep to convert from the ROS dependency name into the matching system package name.
6. Once all dependencies have been registered in the installers and the full transitive closure has been determined, we apply a blacklist to the system package list since we don’t want to install libc*, gazebo*, and other packages provided by our infrastructure. If you want to override the blacklist behavior, you can provide your own blacklist with the --apt-package-blacklist argument.
7. After blacklisting, we invoke install() on all the registered BundleInstallerExtensionPoints that have been used during the dependency registration step. The installers install everything into a directory within the --bundle-base (default: bundle) directory. Once the downloading and installation is complete, we copy in our setup.sh, which enables overlaying, and then tar and gzip the folder into dependencies.tar.gz.
8. After the dependencies have been archived, we archive the latest built install folder. We tar and gzip the entire --install-base (default: install) directory into a separate overlay with a setup.sh to enable overlaying.
9. After both archives have been created, we collect all the metadata that has been produced during the bundling process and add metadata describing the size, offset, and ordering of the overlays. We then tar and compress that metadata into metadata.tar.gz.
10. We then tar the various archives together to create the ‘bundle’ and write it to disk at bundle/output.tar. The bundle is now usable locally on on AWS RoboMaker.

Using a Bundle to Run an Application

The bundle format can be used locally just like we use it on AWS RoboMaker. Doing this could be useful in your application, or just as a debugging tool. We will soon publish the GoLang library we have built to consume colcon bundle archives in our services. For now, I’ll summarize the steps of how we currently read and consume a colcon bundle archive:

Wrapping Up

We are happy with latest the latest iteration of our format and tooling, and we look forward to continuing to improve upon what we have. The colcon bundle implementation is extensible to more package types and installers. If you’re interested in extending the functionality, we welcome issues and contributions incolcon-bundle and colcon-ros-bundle. If you want to build your own tools on top ofcolcon, check out the colcon contribution documentation.