The Environment Cookbook Pattern

Back in August of 2012 I appeared on the FoodFight show to discuss emerging development patterns which inspired the creation of Berkshelf - A cookbook dependency resolver and development workflow assistant. Three of the patterns I mentioned were The Application Cookbook, The Library Cookbook, and The Wrapper Cookbook. The contents of the episode were also summed up well, but with few inaccuracies (definitely due to my poor verbal articulation skills), in an awesome blog post by Bryan Barry which then also became a central part of the ChefConf 2013 talk titled: The Berkshelf Way.

That was over a year ago. Berkshelf had just received a 1.0 release and proper cookbook patterns were still emerging. When I was writing the content for the ChefConf 2012 talk, the Application, Library, and Wrapper cookbook patterns were already really clear. However, during that time it was very clear that there were still additional patterns that we had not yet discovered or just not yet named.

With the Library, Application, and Wrapper patterns we had a way to effectively work in isolation, distribute, and reuse each other's cookbooks, but there was still a problem of promoting an application throughout various environments in the development cycle. Two cookbooks could potentially live on the same node causing a dependency conflict, or worse they could NOT raise a dependency conflict and a cookbook you hadn't expected to be activated would be used by Chef Client. These problems can be solved by something I call The Environment Cookbook.

Before we get to the Environment Cookbook, let's quickly recap what the well known other cookbook patterns are.

You may want to read this even if you believe you are familiar with these patterns! The vocabulary used to describe these patterns is often swapped or incorrect.

The Library Cookbook

This is the most basic building block of Cookbooks. These types of cookbooks are reusable and are mixed into other cookbooks to enhance them by:

  • Adding LWRPs that abstract common functionality
  • Including Libraries that add Ruby modules/classes for any depending cookbooks

The goal of these cookbooks is to abstract common things into re-usable building blocks. They often do not include a single recipe because their job is to solely enhance the Chef primitives. It is also very common for these cookbooks to not include attributes since there's nothing to configure.

Examples of some Library Cookbooks:

  • database-cookbook - a cookbook containing LWRPs for creating databases, running SQL queries, creating database users, and other useful database things.
  • github-cookbook - a cookbook that provides an LWRP for deploying out of the Github releases API.
  • libarchive-cookbook - a cookbook that provides an LWRP for extracting archive files.
  • artifact-cookbook - a cookbook that deploys various archive types from various artifact storage locations.

Library cookbooks may depend on other library cookbooks or application cookbooks. They never depend on an Environment Cookbook and they never depend on a Wrapper cookbook.

I often hear people call the NGINX, Apache, or MySQL cookbook a library cookbook. These are not library cookbooks, they are application cookbooks. While these cookbooks may provide a useful LWRP, they still perform the job of installing a piece of software and managing it.

Since the name of an LWRP is derived from the cookbook it is defined in, these cookbooks are named with that in mind. Pick names that make sense for the way you want LWRPs to appear in recipes.

Every Library cookbook should live in it's own version control repository.

The Application Cookbook

These cookbooks are a level above Library cookbooks. They always contain at least one recipe (the default recipe) to install a particular piece of software sharing the same name as the cookbook. If the application the cookbook manages contains multiple components then each one is broken up into it's own recipe and the recipe is named after the component it will install. Things are broken up in this way so you could install various components spread across a number of nodes within an environment.

These cookbooks almost always contain a set of attributes which act as the runtime configuration for the cookbook. These attributes can do something like setting a port number or even describing the desired state of a service.

Examples of Application Cookbooks:

  • berkshelf-cookbook - used for deploying a Berkshelf-API server. Will also contain a recipe for installing and configuring Berkshelf itself in the near future.
  • mysql-cookbook - installs and configures MySQL
  • nginx-cookbook - installs and configures NGINX

Application cookbooks may depend on Library Cookbooks and other Application Cookbooks. They never depend on Environment Cookbooks. They never depend on a Wrapper or Base Cookbook unless they are intended to be internal to your organization and will never be distributed to the Chef Community Site.

These cookbooks are always named after the application they manage.

Every Application cookbook should live in it's own version control repository.

The Wrapper Cookbook

This is the lightest Cookbook out of all the known Cookbook patterns. It does a very simple job of depending on an Application Cookbook and then exposing a recipe for each recipe found in the Application Cookbook that it is wrapping. In these recipes a single call to include_recipe "{wrapped-cookbook}::{wrapped-recipe}" will be found along with a number of node.set[] functions which override the default values of the wrapped Cookbook. It looks something like this:

#
# Cookbook Name:: vialstudios-postgresql
# Recipe:: server
#
# Author:: Jamie Winsor (<jamie@vialstudios.com>)
#

node.set[:postgresql][:config][:port]                             = 4500  
node.set[:mysql][:config][:max_connections] = 150

include_recipe "postgresql::server"  

No links to examples because these cookbooks aren't contained in public Github repositories because they are very often specific to an organization's needs.

Wrapper cookbooks depend on Application Cookbooks only. They do not depend on other Wrapper Cookbooks, Library Cookbooks, or Environment Cookbooks.

These cookbooks follow the naming convention {organization}-{wrapped_cookbook} or even sometimes {application}-{wrapped_cookbook}. So the postgresql cookbook for the company Vialstudios would be called vialstudios-postgresql.

Every Wrapper cookbook should live in it's own version control repository.

The Base Cookbook

Each organization should have one of these. This cookbook does the job of setting the MOTD on your machines or creating users and setting them up with zsh instead of bash. This cookbook can become a sort of "junk drawer" so you should be careful when adding to it. Things should only be added here when it doesn't make sense to place them in another spot.

This is another cookbook pattern that you don't see out in the wild because it's specific to your organization and shouldn't be shared with anyone else.

Base Cookbooks may depend on Library Cookbooks, Application Cookbooks, or Wrapper Cookbooks. They never depend on an Environment Cookbook.

These cookbooks follow the naming convention {organization}-base.

Every Base cookbook should live in it's own version control repository.

The Environment Cookbook

This is the piece that ties the release process of your development cycle together and allows you to release software that is easy to install and to configure in anyone's infrastructure as long as they have a Chef Server.

An Environment Cookbook is nearly identical to an Application Cookbook in structure with one seemingly small, but crucial difference: The Environment Cookbook is the only cookbook which should have it's Berksfile.lock committed into version control.

Documentation of an Environment Cookbook should also be top priority. Special care should be taken to fill out the metadata to include:

  • A depends entry for each cookbook that you will want to place in this environment. Be sure to include your "base" cookbook if you have one.
  • A recipe entry for each Public Recipe containing a useful description. A Public Recipe is one that can be placed within the Run List of a node.
  • An 'attribute' entry for each Public Attribute containing a useful description, a default value, a type, etc. A Public Attribute is one that is exposed as a configurable option to the end user.

Heres a sample metadata of an Environment Cookbook:

name "my_face"  
version File.read(File.join(File.dirname(__FILE__), "..", "VERSION"))

depends "vialstudios-base"  
depends "nginx"  
depends "postgres"  
depends "rbenv"

recipe "my_face::default", "Install a full stack MyFace server."  
recipe "my_face::app_proxy", "Install a MyFace application server proxy."  
recipe "my_face::app_server", "Install a MyFace application server."  
recipe "my_face::database_server", "Install a MyFace database server."

attribute "my_face/app_server/port"  
 display_name: "App Server Port",
 description: "Listen Port for application server",
 type: "integer",
 default: 10100

attribute "my_face/app_proxy/port"  
 display_name: "App Proxy Port",
 description: "Listen Port for application server proxy",
 type: "integer",
 default: 80

And the Berksfile looks something like this:

source "https://berks-api.vialstudios.com"  
source "https://api.berkshelf.com"

metadata  

Running berks install will now produce a Berksfile.lock for us which should be committed into version control. We will apply this Lockfile directly to an environment on our Chef Server to ensure that every single node which ever joins that particular environment is guaranteed to have the exact cookbook versions we expected and tested against.

It is very common for an Environment Cookbook to depend on Wrapper and Base Cookbooks and, just like an Application Cookbook which depends on a Wrapper Cookbook, don't distribute an Environment Cookbook with one of these dependencies to the Chef Community Site.

Distributing and Using Environment Cookbooks

Once we have our cookbook we now need a way to generate an artifact that we can ship alongside our software. It should contain everything you need to create new environments or add-on to existing ones.

Berkshelf 3.0 contains an improved berks package. It will package all of your Berksfile's dependencies into a tar.gz archive in Chef-Solo friendly format. It will also include a Berksfile.lock next to the cookbooks directory inside the archive.

With the Berksfile.lock that is contained in our archive we can run the berks apply <environment-name> command to apply a strict, and 100% accurate constraint list to a given environment.

The Chef Environments that an Environment Cookbook manages should follow the naming convention {cookbook_name}-{environment_name} where {cookbook_name} matches that of the Environment Cookbook and {environment_name} is what you would have probably called this environment in the past. Something like dev, staging, preview, or production.

We should also use this Chef Environment as a central place for configuration. All of the configurable options that we exposed in the metadata through attribute entries are tunable on this environment and act as a configuration screen for our application (you can even go the distance and make a small dashboard for this that reads your cookbook's metadata to expose configurable fields!).

Versioning

Unlike the other Cookbook patterns, I don't recommend keeping an Environment Cookbook in it's own version control repository. I find it best to place this cookbook alongside the actual source of your application. This will allow you to keep the version number of the Environment Cookbook in lock step with your application (provided your application also follows SemVer).

Wrap Up

The Environment Cookbook pattern worked so well for me that over the last two months I have been making changes to some of the behaviours of Berkshelf's pre-existing commands to make it even easier to adopt these patterns. Look forward to a Berkshelf 3.0 release early next year but feel free to read this small set of instructions on how to use the master branch right now and send us your feedback.

I hope this article helps clear up some confusion around the pre-existing cookbook patterns and also sheds some light on some of the new emerging stuff.