Doing software development with a Continuous Integration process involves a lot of automated testing. Doing tests consumes precious time so we decided to do some tinkering to make our test suite load and execute faster.

As your test suite grows, it takes longer to run the tests, and soon you have a problem because your test suite is running for over 10 minutes. Now you have two options: reduce the number of tests or optimize the test execution time.

A few months ago, we wrote about how we do Continuous Integration in Ruby on Rails. Back then, a test suite for one of our projects (around 1400 tests) ran for about 7 minutes. Today, an even larger test suite (around 1700 tests) runs for 2 and a half minutes. How did we do it?

Every time you call a rake task, your Rails application has to be loaded. The time it takes to load the application depends heavily on the size of the application and the number of gems the application is using. In our example, it was around 10 seconds. At first, 10 seconds doesn’t sound that bad, but when you realize you are calling rake tasks all the time and you have to wait for 10 seconds every time, you start to feel unproductive.

So we started using Zeus. Zeus speeds up loading time of your Rails application by preloading your application only once and then monitoring file changes in your application directory and doing reloads in the background. Now you can call a rake task with: Zeus rake task and it’ll take less than one second to start because the Rails application is already preloaded.

Running a large test suite with Zeus isn’t a big win (we cut off the first 10 seconds). It’s most useful when you’re doing TDD because you run your tests numerous times while developing. If you have to wait 10 seconds every time, it’s just not doable.

Just take this simple model spec with 9 tests as an example:

With the latest release of Rails 4.1, the Spring application preloader was introduced as a part of the standard Rails environment. The idea behind Spring is the same as behind the mighty Zeus – preload your Rails application to speed up development. We tried using Spring on our projects and it works almost equally awesome. We favor Zeus primarily because it’s noticeably faster (roughly 30% faster), but we are eagerly awaiting any future reversals.

The thing that actually makes a huge difference when running a large test suite is running tests in parallel. Modern CPUs have lots of cores, which makes utilizing those cores for testing very effective.

Luckily, there is a gem called Parallel Tests which does just that. The gem creates multiple test databases (one for each core) and then distributes all tests across those cores and runs them in parallel.

You can even combine parallel tests with Zeus, using a gem called Zeus Parallel Tests. With this combination, we reduced the running time of our test suite by over 50%.

When a test is slow, it probably means that the model you’re testing is not optimized and you should be thinking about refactoring.

A typical antipattern is creating fat models with a lot of business logic and then wondering why tests are slow. It’s because you are testing things which don’t need to be tested (e.g. ActiveRecord) but mocked, and you’re needlessly hitting the database.

Let’s show this using a simple example:

Now let’s try to refactor our model and extract that #total_price functionality to a separate class so we can test it in isolation:

Now we don’t need to require the spec_helper (which loads Rails), we can mock the items and avoid hitting the database.

We like this approach, not only because our tests are faster, but also because it drives us to write better code and avoid writing fat models.

One important thing to remember is – don’t let your tests access any external web services. There are several reasons why this is bad:

This can be solved by disabling all remote connections in your tests, mocking the external service functionality and testing it in isolation. We use the WebMock gem for this.

When you work on applications that your clients are actively using, you want to be 100% sure you don’t break something when deploying a new feature.

We all know the feeling when your ticketing system starts screaming at you with bug reports because you’ve deployed some buggy code and your application has started to misbehave. With that in mind, I’ll show you how our Ruby on Rails development team handles this with testing and a Continuous Integration process.

Continuous Integration (CI) is a software development practice where members of a team integrate their work as frequently as possible to reduce integration problems and code conflicts. Every push to a source code repository is verified by an automated build and a test suite to detect errors as quickly as possible. A good CI tool also has a good notification system which effectively notifies all the team members about conflicts and issues with their code.

First of all, to get started with any sort of Continuous Integration – you need to write tests. Without a test suite, it’s impossible to check if changes to a codebase broke something; well, at least in an automated way. You can always manually open every view, every form, and try every action in your application after you change something, but that’s hardly an efficient way to do it.

We mostly write unit tests and test model validations, scopes, callbacks, associations and complex methods to be sure the model is behaving as it should. We write functional tests to check controller actions, filters, redirects, view rendering, and the like.

Functional tests also help us to approximately check for errors in our view files. If we have an error inside a view file, the controller test using that view file will fail while rendering it, and we’ll easily find the source of the error.

We use several gems to help us in writing the tests:

Some applications have a lot of code in JavaScript, so we started with some basic JavaScript testing for Rails applications. Here we’re using:

By using jasmine-rails, we’re able to run the JavaScript specs either in the browser or from the command line. The latter is important for us because it enables us to automate this task.

When we have a good test suite, we want to further automate our workflow. You can go two ways here – set up and maintain your own CI server or go with a hosted one and just pay someone else to do it for you.

We went with a great hosted CI solution called Semaphore. Semaphore connects to your Github repositories and receives an automatic trigger for every code commit. From there it runs standard build tasks like bundle install, rake db:migrate, rake test, etc. All these tasks can be modified to suite the needs of the project.

If the build fails, notifications get sent out to team members, who can easily track the problem in the build log. This way, our team doesn’t need to waste time on running all the tests locally. They simply commit the code and get notified only if something goes wrong.

If all tasks run successfully, it can also do automatic deployment to a server you define. We usually have two servers for every project:

When we’re developing new features, we usually do them in feature branches so we don’t mess with the master branch. The master branch needs to always be “deployable”. When we create a new branch, it gets automatically added to Semaphore.

We use one more tool in our workflow. It’s not a standard CI tool, but a tool for automated code reviewing, called CodeClimate. It also gets triggered on every code commit and runs a series of code quality checks and security analyses like:

With CodeClimate, our team gets notified about new issues and changes in code quality in real time. This increases the team’s awareness of the state of the code and improves overall maintainability in the long run.

We’ve also integrated CodeClimate with HipChat so we receive notifications directly to HipChat rooms. Since we use Hipchat as our primary communication tool, the fact that Semaphore and CodeClimate are both integrated with it comes in really handy.

You see, when you’re developing a complex application, chances are you start getting sloppy. You cut corners, introduce hacks and, in general, stop paying attention to code quality. CodeClimate keeps us in check and stops us from incurring too much technical debt.

With the combination of different tools and techniques mentioned before, we’ve managed to automate our workflow and drastically reduce the time we spend on testing, building, deploying and debugging our applications. As always, there is still room for improvement, but we are proud of what we’ve managed to achieve so far.

Please note that it’s very important to have a dedicated Continuous Integration server. The reason for that is simple – the nag factor! The CI server, especially if it is integrated with HipChat or a similar communication tool, constantly nags developers to solve the issues that show up with their code. This way, problems get solved as they are introduced – every hour, every day – which is exactly the point of Continuous Integration.