Continuous Integration on Android: Why We Ditched Jenkins for Circle CI

In our Android development team, we use the Navy Seals Android Development Process. It includes getting automatic feedback from static analysis tools and running tests using a Continuous Integration server. While Jenkins is a solid tool, it has its quirks. So when Circle CI announced Android support, we decided to give it a spin.

We used Jenkins CI for a while in combination with Github and Bitbucket repositories. Jenkins was installed on a local machine in the company and each time someone pushed to a repository, a web-hook triggered a build. Then Jenkins fetched the current code, built it, ran static analysis and our tests. If there was any problem at all, it notified us via the appropriate Slack channel.

That sounds pretty straightforward, right? Unfortunately, there’s a big difference between how things should work and how they actually work.

Jenkins is a mature piece of software and has a large number of available plugins to make your life easier. However, the UI looks really ugly and a large percentage of plugins do not work as advertised.

This problem is not specific only to Jenkins, but to any self-hosted CI solution. Somebody needs to ensure that the machine is secure because it contains a lot of code that is core to our business. This includes securing access to the machine and regularly updating to patch the latest OpenSSL vulnerabilities.

Security issues are not the only issues you might encounter. In our case, builds started failing across all projects at one point. The disk was full because of all the archived build artifacts and the builds kept failing until somebody found the time to clean up the disk. We ended up buying a larger hard drive to accommodate more projects.

These are not unsolvable or even difficult issues, but they take time to solve – time that we could otherwise spend improving our apps.

In our Jenkins setup, you install all the tools you need to build the app locally on the machine. They are used in the build script when your build is run (think build tools, Android SDK, support libraries, etc.). And when you update your code so that it needs a new version of a tool, you also have to remember to log onto the Jenkins machine and manually install the new version. If you don’t – your build fails.

About ten months ago, Circle CI announced support for Android and iOS projects. Pretty soon, we started trying it out since we weren’t satisfied with our setup. We discovered that it had some great advantages over our old Jenkins setup.

Each time a build is triggered, a new virtual container is spun up. The project GitHub repository is cloned and the circle.yml configuration file in the root of the project is read and used in the rest of the process.

The configuration file is used to:

Unlike Jenkins, which has a rudimentary and cumbersome UI, Circle has obviously put a lot of work into their Web UI because it’s a breeze to set up and use. Circle CI lists all your GitHub repositories so you can set up CI for your project in just a few clicks. It also automatically adds the deploy key to the repository.

Mail, Slack, Hipchat, IRC.. you name it. There’s also an option to notify you only if the build status changes.

The first container is free, an additional container (two parallel builds in total) is $50 per month. You can compare that to Travis CI, where the lowest tier is $129 per month. For a smaller project, you could probably get away with one container.

All the configuration needed for Circle to run your build is in circle.yml inside your project. When you need to update your build tools, you do it right inside of your project. When you push your code, you do it knowing that the new build tools will be installed before your build is run. Also, if you want to retry any older build, you don’t have to worry about not having the older build tools installed.

When you create a pull request, GitHub displays the Circle CI build status. You can also tell GitHub to disallow merging the pull request if the build was not successful.

Circle CI does not support KVM emulation. This means you can’t use x86 emulators like Genymotion and you’re stuck with arm emulators which take forever to boot. It also needs a small amount of time to spin up a new container – but that’s negligible next to the emulator boot time!

Although we initially experimented with running Android instrumentation tests on the emulator, we are now using Robolectric unit tests and are quite happy with them.

Our typical build on a project with 130 tests takes 14 minutes, but if you’re using the Android emulator, it will take more than 20 minutes.

Circle CI has great GitHub integration. I especially like that it indicates if the tests passed or not for each pull request, so I don’t even need to look at the code if the tests fail. However, there is absolutely no Bitbucket support. As far as I know from speaking to their tech support – there are no plans to implement it in the near future.

We’re still hoping Circle CI might add Bitbucket support one day. For the time being, we’ve relocated some of our Bitbucket repositories to GitHub.

When you use a hosted CI solution, you need to be aware that your code will be downloaded to a server that you do not control. If a security breach should happen on that server, your code may be compromised. If your company has strict security policies, you may not be able to use Circle CI.

There are several commercial CI solutions that support Android, like Travis and Ship.io. However, we’ve found Circle CI to best fit our needs. Travis Android support is still in beta and we haven’t found a simple way to store build artifacts with it. At the time when we were looking for a Jenkins replacement, Ship.io was still in public beta and we liked the Circle CI pricing model better.

I’m sure time did not stand still for Travis and Ship.io, so it might be a good idea to take another look at them. There are also some hosted Jenkins solutions out there if that fits your use case.

Update: Ship.io is shutting down.

The problems we had with Jenkins:

Circle CI solves all these problems for us.

The bottom line is that while any kind of CI solution which helps you code is better than nothing, some are better than others. We found that Circle CI is a better fit than Jenkins for our development process, but something else may work better for a different company.

Thanks to my colleagues Ivan Kocijan and Željko Plesac for reading drafts of this post and giving great feedback!

If you are interested in how our Rails team approaches Continuous Integration and delivery, you can read about it in this article.

Are you using Continuous Integration in your projects? What kind of tools do you use? Let us know in the comments!