For some time now I’ve been running Docker-based dev and prod environments on Amazon EC2. Even though Docker is pretty new it works well and it’s a good platform for deploying microservice based applications. Each microservice is packaged as a Docker container, which provides a fast and convenient packaging and deployment mechanism regardless of which language and framework the microservice uses.

Over the weekend I finally upgraded my  EC2 instances to use the latest version of Docker. I’d been meaning to do this for a while but there was a big obstacle: the EC2 instance running the Jenkins-based deployment pipeline was handcrafted :-(. I should have known better. It’s just that this particular server was created when I was getting started with Docker and so I had done a lot of configuration via the command line. Oops!

Upgrading to Docker consisted of creating some Chef cookbooks and a series of Packer templates that built AMIs using those cookbooks. I wrote three different Packer templates for building three different AMIs.

1. The first Packer template built a base AMI containing the newest Docker, Chef Solo and some other utilities.
2. The second  Packer template for extended the base AMI to run a private Docker registry backed by S3. Originally, the registry was part of the base AMI but it took a few iterations to get cookbook for configuring the registry to work. To avoid wasting time repeatedly reinstalling Docker etc.  I moved the registry into it’s own Packer template. In particular, it seems that a Docker container that’s running when an AMI is created still has the old network configuration in an instance created from that AMI. The solution was to remove the container before creating the AMI and then restart it when the instance boots.
3. The third Packer template extended AMI #2 to run Docker containers for various development tools: Jenkins, etc. This template configured the AMI using some Chef cookbooks that were created by reverse engineering the configuration I had done manually.

Since Jenkins home was on an EBS volume, it was trivial to shutdown the old CI server and attach the EBS volume to the newly created CI server. I then kicked off the deployment pipelines for all of the microservices in order to push their containers to the new S3-backed private registry.

Rebuilding prod servers starting from the Docker registry AMI was easy since I’d already written Chef cookbooks to do that. Moving the applications over simply consisted of pulling the containers from the private registry and launching them.

Now that I’ve automated AMI creation future upgrades should be trivial.

Write 100 times “I must not configure EC2 instances by hand”:

I must not configure EC2 instances by hand

I must not configure EC2 instances by hand

I must not configure EC2 instances by hand

….