Matthias Marschall wrote a great peice last week on the pitfalls of making manual changes to production systems. TL,DR; Making manual changes in the heat of the moment will bite you at the most inopportune times.
The article finishes with this suggestion:
You should have your configuration management tool (like Puppet or Chef) setup so that you can try out possible solutions without having to go in and do it manually.
In my experience, this is the key to solving the problem.
Rather than coercing people to follow a “no manual changes” policy, you make the incentives for making changes with automation better than for making changes manually.
Specifically:
- Make it simple. Reduce the number of steps to make the change with automation. It should be quicker to find the place in your Chef or Puppet code and deploy than logging into the box, editing a file, and restarting a service.
- Make it fast. The time from thinking about the change to the change being applied should be shorter with automation than by doing it manually.
- Make it safe. Provide a rollback mechanism for changes. A safety harness can be as simple as a thin process around “git revert” + deploy.
It’s a perfect example of how tools should complement culture.
