How to Package Cookbook Dependencies Locally with Berkshelf

When AWS OpsWorks added support for Chef 11.10, OpsWorks introduced built-in support for Berkshelf, a dependency manager for Chef cookbooks. Enabling Berkshelf installs it on instances managed by OpsWorks so that it resolves cookbook dependencies on the instances as specified in a Berksfile.

Despite the benefits, using Berkshelf with OpsWorks this way does have some drawbacks:

·       OpsWorks doesn’t provide pre-built assets for all versions of Berkshelf. Building Berkshelf on the fly is likely to fail on smaller instances. So you’re effectively forced to use one of the pre-built versions OpsWorks provides.

·       If Berkshelf fails to execute due to a problem in your cookbook, Chef fails to run. You want to avoid this, especially for unattended instance starts, liked those triggered by AutoScaling.

·       Even if your cookbook is fine, Berkshelf might fail on your instance because it can’t retrieve one of the cookbook dependencies.

In this post I describe how to work around all of these shortcomings by bundling a cookbook along with all its dependencies ahead of time. This can be done on your workstation, with any version of Berkshelf. More important, this happens at development time, when it’s easy to respond to problems, instead of at deployment time.

The easiest way to install Berkshelf and many other Chef tools on a workstation is by using the ChefDK, which is an assortment of tools built by the Chef community. Create a Chef cookbook with the chef command-line tool, which is also included in the ChefDK:

chef generate cookbook “server-app”

This command generates a cookbook and a Berksfile, and places both in a folder that has the same name as the cookbook, in this example, server-app. Out of the box, this cookbook has no dependencies. Let’s change that. In the server-app directory, execute the following command:

echo ‘depends “java”‘ >> metadata.rb

Now, your new server-app cookbook depends on the java cookbook.

At this point, you could use the server-app cookbook with OpsWorks the traditional way. To do this, you would create an archive of the cookbook directory and upload it to, say, Amazon S3. You would use this as the cookbook source in an OpsWorks stack, and turn on Berkshelf support. When OpsWorks checks out custom cookbooks, it would detect that Berkshelf support is turned on, install Berkshelf, and download all cookbook dependencies, in this case, just the java cookbook.

But that’s not the approach I want to show here. So instead of uploading the cookbook as is, let’s run the berks package command to resolve those dependencies ahead of time and wrap all cookbooks in an archive:

berks package

This creates a file named cookbooks-1234567890.tar.gz where 1234567890 is an example of a Unix timestamp. This file contains both your own cookbook and the java cookbook. At the time of this post, the java cookbook does not declare any dependencies. If it did, those dependencies also would be resolved, and all cookbooks would be bundled up in your archive.

Now follow the steps I outlined earlier, after we added the dependency on the java cookbook: Upload this cookbook to Amazon S3 and use it as the cookbook source in an OpsWorks stack. But this time, you won’t have to turn on Berkshelf support. As a result, Berkshelf will not be installed on your instances, and no cookbook dependencies have to be resolved at Chef runtime. Because Berkshelf is only ever run on your workstation, you can use any Berkshelf version you want.

Read more about using Berkshelf to package cookbook dependencies locally on our documentation page.