Chapter 7 - DevOps and Deployment

By John Lenz. .

DevOps is one of the new buzzwords and therefore brings with it a lot of fluff and hype, but DevOps has a kernel of great ideas: infrastructure as code, immutable infrastructure, and "Treat servers like cattle, not pets". Infrastructure as code is the idea that you should never manually run commands directly on servers or configure them individually; everything should be described and controlled by files checked into your source repository. Immutable infrastructure is the extension where you move away from shell scripts which edit existing infrastructure. The core idea is that in your source repository you describe what you want the final infrastructure to look like and then say: make it so!

There are a huge range of options for infrastructure: everything from running Kubernetes on your own hardware, to cloud providers, to a fully managed system like Heroku. Once I started using infrastructure as code I quickly realized it is shockingly easy to deploy infrastructure and even quite easy to write the deployment code once you learn. Cloud providers are a natural fit since with the infrastructure automated you create and destroy resources such as compute instances, storage, and databases as needed. I chose AWS. I think of AWS as the swiss army knife of cloud providers; AWS provides a huge range of tools and resources and lets you combine them however you want, creating or destroying compute, storage, and services on demand. This lets AWS live in a sweet spot of large flexibility and infrastructure automation, making it much simpler than running or renting your own hardware and almost as straightforward as a service like Heroku. Google's cloud was a close second, but at the moment they don't have a great way of running Haskell (App Engine has a beta version using docker which might work in the future). I plan on keeping Google's cloud in mind.

The next step is automation. All of the cloud providers allow you to create resources by clicking around their website. Want a Redis cluster? Log in to the AWS ElastiCache developer console website, enter a few text fields like how many servers and how much memory, and click the start button. This goes against infrastructure as code, makes it difficult to replicate a development environment and production environment, and makes it hard to manage changes and updates. Thankfully, AWS (and other cloud providers) allow everything to be done via APIs. There are a large number of tools which take advantage of these APIs to automate infrastructure, and this is currently a space experiencing rapid growth with a new tool or service coming out what seems like every other week (as of late 2016 at least).

Containers

Containers are another buzzword, but at their core containers provide immutable infrastructure. You package the app or service into a container, the container itself is never edited, and the container can be deployed automatically on a cluster of servers. I think of a container as the ultimate output of the build, test, and integrate stages. There are many kinds of containers:

Deployment Automation

Once you have your container, you want to automatically deploy it. If we think of a container as the output of the build of a specific version, then deployment automation will automatically take the container and run it on some servers. This could be uploading a zip file to AWS Lambda, deploying the docker container on a fleet of servers using Kubernetes or ElasticBeanstalk, or starting a new Auto Scaling Group from the AMI.

The oldest tools are orchestration tools such as Puppet, Chef, Ansible, SaltStack, AWS OpsWorks, AWS CodeDeploy, and many others. These tools help run shell scripts and update configuration on a large number of servers at once, and can be used to deploy any of the above kinds of containers. You could provision a machine image, copy the application as a zip file into the image, and perform other configuration at runtime. Or the scripts can start a docker container on a number of servers. The shell scripts you write can become quite complicated and they focus on modifying, provisioning, and editing existing servers instead of immutable infrastructure, so I have avoided these tools.

The alternative is to use dedicated tools to provision infrastructure. The idea is you describe in a configuration file (typically YAML) in your source repository what you want the end result to look like, and then run a tool which compares the configuration file to the existing infrastructure and then either creates new resources or edits existing resources. A good example of this is stack for building Haskell code. In stack.yaml and the cabal file, you describe in a config file what the dependencies are, what files to build, what options to use, etc. and then say stack build or stack test which handles all the details on how to actually perform the build or test. Similarly, you can describe in a configuration file that you want these DNS records, a Redis cluster of this size, a storage bucket with this access policy, this bundle of code running on Lambda, etc. and the tool makes it so. It can also keep the infrastructure alive across failures such as when instances fail health checks, the instance is destroyed a new instance is created.

There are several tools for this, including AWS CloudFormation, Terraform, AWS SAM, and stratosphere for a Haskell EDSL. For Docker, Kubernetes is quickly becoming the biggest Docker management tool. With these tools, you specify something such as "I want 5 copies of this container running behind a load balancer" or "I want a database table with these columns" in a configuration file and then a call to Kubernetes' kubectl or CloudFormation's CLI tool or Terraform makes it happen. There are many other tools such as Apex and Serverless for Lambda and Openshift and Deis for containers. I use CloudFormation, although I might use stratosphere to get a Haskell EDSL.

Pipelines

The source code plus configuration files such as stack.yaml and CloudFormation YAML files are in your source repository. To get the application deployed, the code flows through a sequence or pipeline of stages. Each stage is a call to one of these provisioning tools which take the configuration file from the source and performs the actions. Typically, there will be a build stage, a test stage, a integrate stage (which creates the container), an acceptance test stage, various deployment stages including manual approval stages. Pipeline stages can even do fancy things such as watch the logs for 2 minutes checking for 500 error codes and if a large number are found automatically roll back to a previous version (called canary testing). There might even be separate pipelines: one pipeline for the application and a separate pipeline for the database.

At least for me, pipelines was quite a different way of thinking about infrastructure. I want to add a new DNS record, so instead of adding it directly or even scripting the addition, I instead add the DNS record into the CloudFormation config file, push the commit, and the pipeline automatically updates the infrastructure to match the config file. The goal is no one ever executes a command manually. Everything is triggered by pushing a changeset (perhaps to a specific branch).

There are a variety of tools to control pipelines. The simplest is actually a Makefile on the developer machine and is where I started. The Makefile does the build with a few calls to stack and docker and handles the infrastructure with calls to CloudFormation using the AWS command line interface or Kubernetes using kubectl. The Makefile is only a handful of lines long since all the hard work is in the stack or CloudFormation configuration files. Other options for pipelines are hosted solutions such as AWS CodePipeline, CodeShip, Bitbucket Pipelines. Finally, Spinnaker is a pipeline and deployment tool which you run yourself. You can also extend continuous integration tools such as Jenkins, TravisCI, or others. The CI tool creates and pushes the container (docker image, AMI, bundle of code) to a storage service (S3, container registry), and then uses a call to Kubernetes or CloudFormation to deploy. The disadvantage of extending CI tools in this way is typically there isn't automatic rollback to a previous version on failure, although CI tools are always improving and I wouldn't be surprised to see them become more involved with deployment and monitoring.

My Setup

As I mentioned above, I am using AWS although I am keeping an eye on Google's Cloud. I have two separate AWS accounts with shared billing: one for development and one for production. By using the same pipelines for both, it is easy to keep development and production as the same environment. To manage the pipelines, I started out with Makefiles and eventually started using AWS's CodePipeline and CodeBuild. I have three pipelines.

As mentioned above, I am using Beanstalk but keeping an eye on Lambda. The pipeline first uses stack to build and run the unit test (either locally using the Makefile or using CodeBuild as a CodePipeline stage). Next, it creates a docker image which is pushed to Amazon's Container Registry. Finally, the deployment happens by running some AWS CLI commands to create a new version in beanstalk.

For the docker image, there is the stack-run image which you can use as a base, but instead I took an approach inspired by the Haskell Web Server in a 5MB Docker Image. Since I am not concerned with getting it as small as absolutely possible, I start with the statically linked busybox+uclibc image which is only 650KiB or so. I then add the binary produced by stack build, all the libraries from running ldd on the binary, plus the gconv libraries. I stole some code from the haskell-scratch repository, but since haskell-scratch was missing a few libraries I needed like libcrypto and libssl, I couldn't use it directly. In any case, it was easy enough to copy and modify the Makefile from haskell-scratch to start from the busybox+uclibc image and include the DLLs that I need.