Building testable infrastructure with Chef, Test Kitchen and ServerSpec

Matthew Valentine-House, a Developer at FutureLearn, discusses how we use tools such as Chef, Test Kitchen and ServerSpec to build testable infrastructure.

Building reliable infrastructure is hard; it always has been. But there are some excellent solutions emerging that can alleviate a lot of the pain, as well as increase your confidence and success rate when building pieces of your infrastructure.

There have been major strides in recent years towards treating infrastructure with the same care and attention as the business logic that runs on it, but it’s only very recently, in my opinion, that we’re getting to a place where building infrastructure can be as as robust and safe as building applications.

Tools such as Chef allow us to treat our infrastructure as code and enable us to consistently and repeatably commission the services that we need. This also enables us to use version control and our revision history to document the progress and the decisions we have to make when making configuration changes.

At FutureLearn we use many of these tools extensively and I’d like to use this post to introduce a few of them and walk you through our approach to building testable infrastructure.

Chef

Chef is a framework that allows you to define your infrastructure as Ruby. You write modular packages of configuration scripts (referred to as “cookbooks” and “recipes” respectively) and you apply a “run list” of recipes to a server or virtual machine, to automate the manual provisioning that would normally be carried out by hand, or using shell scripts.

There are alternatives to Chef – from older tools like CFengine and Puppet to newer entries like Ansible and SaltStack. Each of these systems has a different design philosophy, so if you’re not already using one of them, it’s worth spending some time to evaluate and choose the most appropriate solution for your business.

We use Chef, because it fits our workflow closely and it means we can write our recipes in Ruby, which is the language we are most familiar with.

A typical Chef recipe looks like this:

template 'nginx.conf' do
  path   "#{node['nginx']['dir']}/nginx.conf"
  source 'nginx.conf.erb'
  owner  'root'
  group  node['root_group']
  mode   '0644'
  notifies :reload, 'service[nginx]'
end

template "#{node['nginx']['dir']}/sites-available/default" do
  source 'default-site.erb'
  owner  'root'
  group  node['root_group']
  mode   '0644'
  notifies :reload, 'service[nginx]'
end

nginx_site 'default' do
  enable node['nginx']['default_site_enabled']
end

This is taken from the Nginx community cookbook. It’s part of the recipe that configures the default Nginx configuration. It does five things:

1. Installs the main nginx.conf configuration file, from the nginx.conf.erb template defined in the cookbook.
2. This file is placed in the location defined in the variable node['nginx']['dir'], which defaults in the cookbook to /etc/nginx but can be overridden by any recipe that includes this cookbook in case your infrastructure has different requirements.
3. It makes sure the installed config file has mode 644 and owner root:root. If the file has changed since the last run, it will tell the nginx service that it needs to reload, using whatever notification method your operating system supports by default.
4. Installs the default Nginx site configuration in the sites-available/default file relative to your Nginx default config directory. Again this uses a template in the cookbook. And also with mode 644 and owner root:root.
5. It uses a Custom Chef Resource to build the default Nginx site and trigger whether it is enabled or not, based on the content of the variable node['nginx']['default_site_enabled']. Custom resources can be written in Chef to provide groups of related functionality that are always used together, so that you don’t have to write out the same set of resources every time you use it.

So far so good. But from this small snippet it’s hard to visualise the benefits this can give you over some simple bash server scripts.

One such benefit of these systems is that they provide a framework for writing platform agnostic configuration. For instance, the Chef `package` resource has backends for several different package managers including popular OS package managers like yum, apt, dpkg, portage, pacman, homebrew and even Windows application installers, so it’s easy to write recipes that can be run on many different systems with minimal changes.

While we’re only concerned with Ubuntu at FutureLearn, the multi-platform nature of Chef allows the community to write and publish open source cookbooks that are useful to many people in a wide variety of cases.

The Chef Supermarket is a repository of these cookbooks that we use extensively, as it allows us to pull in recipes for configuring common software and means we don’t have to reinvent the wheel. The aforementioned Nginx cookbook is a great example of a community cookbook that’s useable on a wide variety of systems.

Testing

The result of all of this is that configuring servers is still hard. While tools like Chef allow you to automate and re-use a lot of the code, it’s never going to be completely pain free. So as infrastructure developers, how can we increase our confidence that the code we are writing will achieve what we want, without risking the consistency of our production servers?

One popular technique that has been gaining traction is the idea of automated testing. This has stemmed from Chef’s close ties with the Ruby community and the associated predilection for test driving code using tools like Minitest and Rspec.

There are several different ways of testing Chef cookbooks in regular use, all of them are explained very eloquently in Stephen Nelson-Smith’s excellent book, “Test Driven Infrastructure with Chef.”

I intend to introduce a few of the most popular approaches and give some insight into what, and how, we test at FutureLearn.

ChefSpec

Seth Vargo’s ChefSpec is a unit testing library for Chef. Its DSL is based on RSpec so it should be familiar to most Ruby developers. It uses Chef Solo on the local development workstation to evaluate your recipes and run the tests without actually converging a node. This means you don’t have to spend time bootstrapping VM’s, which helps to ensure ChefSpec has a fast feedback cycle.

We don’t use ChefSpec at FutureLearn, primarily because it’s hard to have complete confidence in your configuration when you aren’t actually running your code against a machine and comparing the state before and after. Unit testing in configuration frameworks like Chef is really more like intention testing – because you’re not running the main converge step, you’re really testing that Chef will attempt to do what you’ve asked it to, and not whether your recipes will successfully run on real hardware.

This is useful if you have a lot of logic in your recipes to test, or you have many different architectures or operating systems to support. Conversely, if you’re only targeting one specific system, and you try to write small, single purpose cookbooks, then you can often feel like you’re repeating yourself: writing your desired outcome using one DSL to test it’s been implemented correctly in another DSL!

At FutureLearn, the approach we take is one of many small, targeted, single purpose cookbooks, which integrate with each other. We like to have a more holistic approach to testing.

Test Kitchen

Integration testing configuration code has always been a slow process, predominantly because it involves building a virtual machine and provisioning it with every test run. And if you’ve been invested in Chef or Puppet for a while, you’ve probably had a lot of experience in writing bootstrap scripts and test harnesses that facilitate this process.

Test Kitchen is a harness that abstracts away most of the hard work and optionally automates most of the integration testing process.

By creating a kitchen.yml file in the root of your Chef cookbook you can configure exactly what hardware you need to run the tests on; how to provision the machine; and which combinations of test suites to exercise on the hardware.

kitchen.yml is divided into several separate sections. The first is the driver and provisioning section. For our simple cases it looks like this:

---
driver:
    name: vagrant

provisioner:
    name: chef_solo

platforms:
    - name: ubuntu-12.04

Because our requirements are simple, we only need to test on Ubuntu 12.04. This part of the configuration builds us a Vagrant VM running Ubuntu 12.04 and provisions it using Chef Solo. This file lives in the root of your cookbook and the machines that are created can be automatically provisioned with the cookbook that you’re testing.

Test Kitchen is very flexible – it has drivers built in for Vagrant, Amazon AWS, DigitalOcean, Docker, LXC containers and many more, and configuring each service is generally just a couple of lines of configuration in this file. We use Vagrant primarily, because it’s the easiest to get started with and it meets our needs adequately.

The provisioner defines how you want your cookbook applied to the Virtual machine. chef_solo uploads your cookbooks to a temporary directory on the virtual machine and runs chef solo to apply them. chef_zero starts up a local in-memory Chef Server that hosts your cookbooks and starts a chef client run from the testing machine. There’s also a shell provisioner so you can use Test Kitchen to integrate test provisioning methods that aren’t Chef. Writing custom providers is also possible – there’s even an open source Puppet provisioner.

It’s a good idea to stick to the provisioning method that matches your production environment most closely. At FutureLearn, our Chef cookbooks are primarily used for bootstrapping AMI’s using Packer so the chef_solo provisioner fits our needs.

The platform’s definition is a list of different systems that you intend to test your cookbook on. There are some nice defaults set up when you use the Vagrant provisioner, so that Test Kitchen can work out what boxfile it needs to use. Other drivers may require more verbose configuration.

The next section in the file is the suites. These are used in case your cookbook contains independent groups of configuration or configuration that needs to be exercised. The suites defined in one of our application cookbooks looks like this:

suites:
    - name: default
      run_list: 'recipe[app::default]'
      attributes:
    - name: queues
      run_list: 'recipe[app::queues]'
      attributes:

This defines two suites (default and queues). Each suite has a different run list, so we can exercise the recipes contained within this cookbook independently. This is a simple example, but for more complex situations, you can also vary the node attributes passed to Chef in each suite. This enables you to test multiple paths through the cookbook or alternative situations that you might encounter.

Running the tests

So how does Test Kitchen run tests based on the kitchen.yml configuration that you give it? There are two ways. The first is using the command kitchen test; this is a one stop solution and is predominantly designed for CI servers. kitchen test carries out the following operations

1. Starts up all required virtual/cloud instances
2. Provisions each machine
3. Uploads and executes the appropriate tests for each suite
4. Reports errors back to the caller
5. Tears down the test instances.

Because this process involves setting up and tearing down up to a handful of machines (we’ll cover this soon), as well as provisioning and testing, it can be slow. This makes it less suitable for fast feedback when writing Cookbooks. My preferred approach for this separates the process out into its individual steps:

Start up all required instances

kitchen create will create a virtual machine instance for every suite and platform combination defined in your kitchen.yml. This is so that each test suite can be run in its own isolated environment and you can have confidence that your recipes are not reliant on each other. Running this for the example above gives an output something like the following:

-----> Starting Kitchen (v1.2.1)
-----> Creating <default-ubuntu-1204>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ...snip...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
       ==> default: Setting hostname...
       ==> default: Machine not provisioning `--no-provision` specified.
       Vagrant instance <default-ubuntu-1204> created.
       Finished creating <default-ubuntu-1204> (0m34.81s).
-----> Creating <queues-ubuntu-1204>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ...snip...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
       ==> default: Setting hostname...
       ==> default: Machine not provisioning `--no-provision` specified.
       Vagrant instance <queues-ubuntu-1204> created.
       Finished creating <queues-ubuntu-1204> (0m21.23s).
-----> Kitchen is finished. (0m57.33s)

I’ve snipped out most of the network setup output for brevity, but Test Kitchen helpfully sets up an SSH server on the instances and forwards port 22 to the local machine. It also configures an authorized SSH key. The consequence of this is that you can run kitchen login with the suite name to ssh directly into the machine should you need to inspect anything manually.

Provision the machine

This is achieved by running kitchen converge. You can converge one suite at a time by passing its name as an argument. Leaving this off converges all of the suites defined in your kitchen.yml.

-----> Starting Kitchen (v1.2.1)
-----> Converging <default-ubuntu-1204>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 3.1.5...
       Removing non-cookbook files before transfer
-----> Installing Chef Omnibus (true)
       downloading https://www.getchef.com/chef/install.sh
         to file /tmp/install.sh
       trying wget...
       ...snip Chef run output...

This step provides a super easy way of running Chef against your new testing VMs. You don’t need to worry about key pairs or running a server – or even manually getting the cookbooks up to the machine. Test Kitchen handles all of that for you. It also integrates with Berkshelf or Librarian-chef to fetch your dependencies.

Run your tests

kitchen verify will verify that the state of the testing machine matches the desired state outlined by your tests. Here’s a simple example of a scaled down test suite that we run against a recipe that sets up some queue processing using Sidekiq:

-----> Starting Kitchen (v1.2.1)
-----> Setting up <queues-ubuntu-1204>...
Fetching: thor-0.19.0.gem (100%)
Successfully installed thor-0.19.0
Fetching: busser-0.6.2.gem (100%)
Successfully installed busser-0.6.2
2 gems installed
-----> Setting up Busser
       Creating BUSSER_ROOT in /tmp/busser
       Creating busser binstub
       Plugin serverspec installed (version 0.5.3)
-----> Running postinstall for serverspec plugin
       Finished setting up <queues-ubuntu-1204> (0m13.89s).
-----> Verifying <queues-ubuntu-1204>...
       Suite path directory /tmp/busser/suites does not exist, skipping.
Uploading /tmp/busser/suites/serverspec/queues_spec.rb (mode=0644)
Uploading /tmp/busser/suites/serverspec/spec_helper.rb (mode=0644)
-----> Running serverspec test suite
-----> Installing Serverspec..
...snip installing Gems...
-----> serverspec installed (version 2.8.1)
/opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -I \
  /tmp/busser/gems/gems/rspec-support-3.2.0/lib:/tmp/busser/gems \
  /gems/rspec-core-3.2.0/lib /opt/chef/embedded/bin/rspec --pattern \
  /tmp/busser/suites/serverspec/\*\*/\*_spec.rb --color --format \
  documentation --default-path /tmp/busser/suites/serverspec

Sidekiq Upstart config files
    Builds a management config to allow all workers to be controlled
    builds a worker config for the data export worker
    builds a worker config for the default worker

Sidekiq worker jobs
    starts the sidekiq manager at boot
    starts the sidekiq default worker at boot
    starts the sidekiq data_export worker at boot

Finished in 0.15976 seconds (files took 0.31117 seconds to load)
6 examples, 0 failures

       Finished verifying <queues-ubuntu-1204> (0m11.94s).
-----> Kitchen is finished. (0m26.93s)

This output is pretty dense, so it’s worth taking some time to step through it.

Busser

Test Kitchen runs tests on the VMs using a Ruby gem called Busser. Busser is a framework for bootstrapping a test suite when you can’t rely on having the correct system dependencies. There are Busser plugins for many popular test frameworks available; Cucumber, Rspec, Minitest, ServerSpec and Bats are all supported.

Which plugin Busser installs will be determined automatically by busser based on the directory structure of the tests in your cookbook. Tests are organised in the following structure

<Cookbook root>/<Suite name>/<Plugin name>/<Test files>

So, as an example, the spec we just ran in the previous example lives at:

cookbook/queues/serverspec/queues_spec.rb

Busser correctly introspects the cookbooks file structure (which has been uploaded on to the VM as part of the previous converge step), to determine that our tests are written using ServerSpec. From there, it can install the ServerSpec plugin, which itself installs ServerSpec and runs our tests through it.

ServerSpec

ServerSpec is a testing library that is heavily influenced by RSpec. It uses the same describe/it patterns in its test definitions and shares a lot of the same expectation syntax, but provides a lot of functionality designed to integrate well with the sort of things you’ll need to test against servers. Let’s take a look at a simple example of a ServerSpec test.

describe 'Sidekiq dependancies' do
  it 'installs Redis' do
    expect(package('redis')).to be_installed
  end
end

describe 'Sidekiq worker jobs' do
  it 'starts the sidekiq manager at boot' do
    expect(service('fl-sidekiq-manager')).to be_enabled
  end
end

From this, we can see that it supports daemonised services and package installation. ServerSpec tests the actual state of your machine, using a command execution library called Specinfra. Specinfra supports over a dozen different operating systems at the time of writing this, and support is always expanding.

In conclusion

At FutureLearn, we use Test Kitchen with Serverspec exclusively for testing our cookbooks. By writing small, single purpose cookbooks and keeping our leaving VMs running between tests, we can keep the feedback cycle as short as possible. This makes it feasible to test drive our infrastructure code thoroughly, without it becoming an exercise in frustration.

The majority of the testing tools we have chosen are mostly agnostic of platform and configuration management tools, so our tests are re-usable and flexible, should we ever make any changes to the way we manage our infrastructure in the future. It is entirely possible to run our ServerSpec tests against a real production machine, purely to check what state it’s in before we run anything against it.

Hopefully this post has outlined some of the many ways it’s possible to test your infrastructure code and shown that getting set up is not as complex as it initially seems. The landscape of infrastructure testing tools is always growing and expanding. Getting started as soon as you can will really help increase your confidence in your recipes, and reduce the amount of time and frustration spent commissioning new machines and environments.

Good luck!

Want to know more about the way we work? Take a look at all of our “Making FutureLearn” posts.