Guide to Application Development Infrastructure: Pairing and CI Servers

This past year, we set up a new pairing server and a new continuous integration (CI) server, because we needed it to be easier to onboard developers and projects, while maintaining project isolation. We used Chef and Capistrano to do this—see my slides for an intro to Chef, “Cook Some Tasty Servers with Chef,” and my slides that take it a little further, “Practical Chef and Capistrano for Your Rails App.”

In this post we’ll walk you through the details of how our application development process benefits from our servers, how we built our pairing server (Bruizer) and our CI server (Borg), how we use them, and what we are looking to improve on. If you’re in a similar position to the one we were in before setting this up, feel free to use this as a jumping off point. Comment and let us know what works for you.

What We Wanted Out of Our Pairing Server, Bruizer
At SmartLogic we often pair program, but we aren’t always in the office at the same time (and everyone is remote every-other Friday). Before we had a pairing server, we SSHed into our own personal machines. The disadvantages of that were:

• People feel privacy concerns about letting other developers into their individual machines.
• Any developer who wanted to host pairing had to have the full project set up on their machine. If the full project is set up on an external pairing server, you can hook into it from any machine.
• When people were using personal machines at home, we had latency problems because of slow home networks. The office network was better suited for low latency application development.

So we started by turning an extra old machine we had lying around into our pairing server and, while it worked, the older hardware made it very slow. We considered using a cloud server, but ultimately felt we could build our own for a reasonable, and one time, cost.

How We Made Bruizer
To create Bruizer, we used Chef to install and configure our application development environment. We continue to use Chef to manage the projects and development dependencies that are available on Bruizer. Chef manages our developers’ public keys and creates accounts for each project, loading the right set of developer public keys in for authentication. Developers can forward their SSH keys to Bruizer so that they can access GitHub as they normally would. We use tmux sessions to share a terminal window, and we use Vim for code editing. For each project, the developers can specify a git repository for the .dotfiles they want loaded, customizing each project to the developers’ needs. We started with a basic set of project dependencies (MySQL, PostgreSQL, Redis, Sphinx, Qt, etc.) and Chef makes adding new dependencies easy.

Hardware Nerdery
For those who like hardware specs, here is what is inside Bruizer and Borg:

• Antec Sonata III 500 Mid Tower Case
• Asus P8Z77-V LK Motherboard
• Intel Core i7-3770K Processor
• 32 GB RAM (2 packs of Corsair Vengeance DDR3 2 x 8GB)
• OCZ Agility 3 256GB SSD
• Ubuntu 12.04 Server

What We Wanted Out of Our CI Server, Borg We created Borg because we are now supporting more projects in production than we were a year ago. A lot of our work was, “build it, hand it off, and we’re done.” Currently, we are maintaining production applications, as well as developing new feature releases for many clients.

At the same time, our test suites have grown large and take a long time to run on a developer’s machine. So it made sense to add CI to our work flow. We evaluated hosted solutions we could pay for, but given the number of production projects that we’re supporting, the one-time capital cost of building our own server made more sense.

How We Made Borg
We ordered the same hardware we had for Bruizer, and used our Chef expertise to install Jenkins, VirtualBox, and Vagrant. We leveraged VirtualBox and Vagrant to provide test isolation for each project’s unique runtime dependencies. We used Jenkins because it’s the gold standard of open source CI server software and it integrated well with Vagrant, which we were already using for many projects. We also use Github for authentication in Jenkins, since all of our projects are already using Github. Nginx is configured to serve up Jenkins on the standard port 80, as well as hosting the Vagrant base boxes needed by our projects (more on this later) so they are accessible for both the automated test process and our developers.

Maintaining the Servers
Maintenance of both Bruizer and Borg is easy with Chef. One common task is keeping the SSH keys of our developers up-to-date. It is a simple change in the Chef data bags and a quick re-cook of the servers and we are done.

Borg requires even less Chef maintenance. On Borg, Chef handled the install of the Jenkins infrastructure, but the actual configuration of Jenkins and projects is manual. The on-going maintenance for Borg is mostly updating the Vagrant boxes that each project uses to execute tests on with new development and testing dependencies. We build these Vagrant boxes with the project dependencies so that they don’t need to be installed with each test run. The goal is that the test script does:

1. bundle install (which shouldn’t actually need to install anything, unless the box is out of date)
2. rake db:schema_load
3. rake

Adding Projects Adding a new project on Bruizer is just a matter of creating a new data bag file describing:

• The list of developers on the team
• The project name
• What shell we want to use
• What version of Ruby from the Ruby Version Manager (RVM) we want
• Whose .dotfiles to load.

A custom Chef recipe we wrote creates the project account and gets the account configured and ready to clone the project code and start hacking.

To add a new project to Borg we:

1. Create a Vagrant box with the project dependencies installed (Ruby version, database server, etc)
2. Place the box on Borg
3. Add a .sls_borg folder to the project to store CI specific configuration (Vagrantfile, database.yml, .rspec). The Vagrantfile will specify the size of the virtual machine (VM), and where to mount the application code within the VM.
4. Then, from the Jenkins web interface, we create a new job for the project that will boot the VM and execute the tests via SSH on the VM.

This all means that the actual execution of project code is isolated to inside the VirtualBox VM, no project specific code is executed directly on Borg’s host OS. We give Jenkins access to private repositories by adding a GitHub user to the project that has Jenkins’ key loaded into it.

Issues
We haven’t had any major problems on Bruizer. The only gotcha is port management for projects. We can’t have everyone spin up Rails processes listening to the same port, so we need to coordinate this on our own. We also wanted to create a Xvfb session for each project with a different display number. We solved this by having the Chef recipe setup the display and export the display number into a .zshrc.local or .bashrc.local file, which assumes the .dotfiles will source this local file.

The biggest issues we have on Borg are surrounding VirtualBox. We often have VMs that won’t start for an unknown reason. We have also seen test runs get hung up during their run and just sit, never timing out, but also never finishing. We can often just restart the build job or forcibly kill a stuck VM, but we have also noticed these issues happening more frequently the longer Borg has been running. A full restart of Borg seems to help resolve the issues for a while. At this point we hope that updates to VirtualBox solve the issue, or we’ll determine a better way to create VM isolation on our hardware. We’ve recently upgraded the version of VirtualBox we were using on Borg from ~4.1 to ~4.2. We have yet to see if this helps our issues with Vagrant/VirtualBox. In order to do this update, we needed the Vagrant plugin for Jenkins to use a newer version of the Vagrant gem, which we applied in a fork of the plugin.

Improvements
We have considered using KVM or OpenStack to provide the hypervisor instead of VirtualBox. We think this might give us better performance or more stability, but it’s hard to justify a change when our Vagrant and VirtualBox setup is so easy to use and is working well enough.

We have recently gotten Jenkins testing Pull Requests on some projects using the pull request ref spec, and we have just started using the Github pull request builder plugin to provide build status feedback right on the pull request.

On the whole, Bruizer and Borg have made our application development process smoother and more stable. What’s your application development environment like, do you use CI servers? Figured out any of the problems we ran into? Comment, and let us know.

Image Source