Why scripting your server installs is a bad idea (and configuration management is awesome)

I was reading the FiveRuns blog post on automating your Rails server configuration, and had a peek at the script they included to setup a machine.

There are some fundamental problems with it.

What happens when the machine can’t reach RubyForge to download the RubyGems tarball? The rest of the script will continue to run and error out horribly, leaving you to wade through a screenful of error output to work out what died.

What about when you need to update your production systems with new packages or configuration files? You’ll probably log into all of them and apply the update (or maybe you’ll write a script to do it). But what about new machines that you roll out? Do you update the install script with the new tasks? Do you use the old install method and then apply the new configuration?

These problems are vaguely manageable with a small number of machines, but as soon as you grow beyond 3 servers it becomes tedious and painful - deployment is hard.

I learnt this the hard way: I started out my sysadmin career building systems like this. The first time I did this I thought it was great. I wrote a ~400 line provisioning script that would set up a PXE booted machine with Samba/Apache/OpenLDAP. This worked exceptionally well in my test environment. Production was another story.

I had inadvertently optimised my deployment for provisioning, not for maintenance.

When you consider the lifecycle of a system, < 0.001% of it’s life is spent being provisioned. The other 99.999% is the server sitting there serving requests, having new software deployed to it, or having it’s configuration tweaked.

Putting all the logic into the provisioning process is setting yourself up for failure. To pull it off, you have to know exactly what the machine is going to be doing for its entire lifecycle. Prescience is not a common trait amongst humans.

So the project requirements change, and the time comes to add another service to the system. How do you manage this? ssh-in-a-for-loop is the obvious response, but then you’d be perpetuating the problem you created for yourself in the first place. And what about new systems? Oh the pain!

House of Cards by Indenture

Unfortunately there’s no easy way out of this mess. You can start from scratch, or push through and waste countless hours of your life maintaining a thousand houses of cards. And you can only start from scratch so many times before it gets equally tedious and fragile.

Configuring your systems with magical scripts is not maintainable. Competent sysadmins around the world have a responsibility to stamp out deployment techniques of this ilk.

This is how some people use Capistrano, and it gives me the willies. Invoking your distro’s package manager from Capistrano is no better than doing it from a shell script. Use the tool for what it’s built for: automating the deployment of your application.

So what do I recommend? Configuration management is a much better solution to this problem. Instead of maintaining separate procedures for provisioning servers and doing change management, you can merge them into one.

Puppet is my configuration management system of choice, and I use it extensively across all the infrastructure I manage. I do this for three reasons:

  1. I forget how machines are configured. I work on a lot of different machines (I touch a minimum of 20 different boxes every week), and it's nigh on impossible to remember how each of them are configured.
  2. Not all operating systems are configured equal. The machines are running a mish-mash of Ubuntu, Debian, Red Hat, CentOS, Fedora, Mac OS X, & OpenBSD, and they're not all on the same version. There's no way i'm going to remember each operating system's eccentricities.
  3. Other people maintain these machines. Someone will login to the machine and change something without telling me. This is a good thing (I am not a single point of failure), as long as we're using some sort of configuration management (I know who to blame when something explodes 6 months down the line (probably me :-)).

Combining Puppet with a distributed version control system is a solid one-two punch to your provisioning and maintenance woes.

Fight for your rights by Frank_BB

Hopefully i’ve convinced you to use Puppet. The first question you probably have is “how do I use puppet to manage $app_built_with_my_framework_of_choice?”.

Just like I recommend not using Capistrano to configure your servers, I recommend using Puppet to manage the infrastructure around your application. Your Puppet manifests should manage web + database + mail servers, monitoring, system package repositories, system utilities, and the standard library of the language your app is written in.

This makes application the side of deployment really simple: for Ruby folk, I recommend bundling all your gems with your application. You should expect the ability to plop your app on any machine that has no more than the Ruby standard library and interpreter and have the app run.

The end result is configuration that can be applied to a new machine in 10 minutes with Puppet, and a seamless app deployment in less than 5 using Cap or Vlad.

When you need to scale further down the line, you can simply apply that same configuration to a new node. Making changes across all your machines becomes an order of magnitude easier - update the manifests, push to your Puppetmaster, have the machines update themselves.

What could be simpler?