Martin Englund posted an open question to the Puppet mailing list a few days ago asking how people are verifying their systems are built as expected:

When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don’t we use that when we install a system?

What are you using to verify that your system is correctly configured and behaves the way you want?

He linked to a blog post demonstrating how he was verifying his machines using Cucumber.

Coincidentally, about a week earlier at Devopsdays in Gent, I was talking to Felix Kronlage and Bernd Ahlers from bytemine about doing similar things through testing SSH and mail delivery with cucumber-nagios.

It’s pretty cool people are thinking about doing BDD/TDD with infrastructure, and it’s even cooler that the tools are at the point where doing this is actually possible.

When doing software testing, your testing tool is normally separate from the language and libraries you’re building the software with (but almost always written in the same language). When testing your infrastructure, I think it makes perfect sense to apply this practice.

So to practise Behaviour Driven Infrastructure right now, you can use Cucumber as the testing tool, and Puppet as the programming language.

One advantage of practicising BDD within sysadmin world is that the testing tools aren’t closely coupled to the language our systems are built with - i.e. if you hate Puppet you can use Cfengine, and if Cucumber isn’t cutting it use PyUnit.

But to something tangible!

Building on Martin’s excellent examples, i’ve pushed out a new version of cucumber-nagios that includes some basic SSH interaction steps, so you can start building behavioural tests for your infrastructure:

Feature: example.org ssh logins
  As a user of example.org
  I need to login remotely

  Scenario: Basic login
    Given I have no public keys set
    Then I can ssh to "example.org" with the following credentials:
     | username | password    |
     | lindsay  | spoonofdoom |

  Scenario: Login to multiple hosts
    Given I have no public keys set
    Then I can ssh to the following hosts with these credentials:
     | hostname           | username | password      |
     | example.org        | matthew  | spladeofpain  |
     | mail.example.org   | john     | forkoffury    |
     | web04.example.org  | steve    | sporkofpork   |

  Scenario: Login with a key
    Given I have the following public keys:
     | keyfile                   |
     | /home/user/.ssh/id_dsa |
    Then I can ssh to the following hosts with these credentials:
     | hostname         | username |
     | example.org      | matthew  |
     | mail.example.org | mark     |

  Scenario: Login with an inline key
    Then I can ssh to the following hosts with these credentials:
     | hostname         | username | keyfile                   |
     | example.org      | luke     | /home/luke/.ssh/id_dsa |
     | mail.example.org | john     | /home/john/.ssh/id_dsa |

The above example shows there’s lots of ways to test the same thing (all depending on what you’re trying to achieve), but there is now also suppport for executing shell commands remotely:

  Scenario: Checking /etc/passwd
    When I ssh to "example.org" with the following credentials:
     | username | password      | keyfile                 |
     | jacob    | spifeofstrife | /home/jacob/.ssh/id_dsa |
    And I run "cat /etc/passwd"
    Then I should see "jacob" in the output

I don’t expect you would do a cat /etc/passwd in a real test, however the step definition is a good example of how to interact with an established SSH connection:

When /^I run "([^\"]*)"$/ do |command|
  @output = @connection.exec!(command)
end

Then /^I should see "([^\"]*)" in the output$/ do |string|
  @output.should =~ /#{string}/
end

You’d use this to write specific tests for checking system behaviour, such as local user logins vs LDAP logins, or the presence of a daemon.

So the resulting process may look something like this:

  1. Use cucumber-nagios to write a specification of how you expect your infrastructure to behave.
  2. Hook your new cucumber-nagios checks into Nagios.
  3. Start writing your manifests/cookbooks.
  4. Run your configuration management tool on the node you’re configuring.
  5. Iterate until your monitoring system is silent.

Not only do you have a functional definition of how your machines work that you can use to build your machines, but if your systems deviate from the expected behaviour at any point in the future, you’ll get an alert from your monitoring system.

Maintaining both a configuration management system and a set of integration tests might get annoying after a while, but if you ever decide to migrate to another configuration management system or move your machines into the cloud you’d have a set of tests you could apply immediately.

This could also be useful for moving existing machines into a configuration management system. Write a set of integration tests for your unmanaged machines, run your configuration management system over the existing machines, see if anything is broken.

I’d be interested to hear how this process or similar works for people!