The Infinum Test Lab: Device Testing

If you want to find bugs in your software before the user does, you need to test on a respectable set of real world devices. In other words – you need a decent test lab. What do you need to build a good test lab? Read on.

Modern software tends to be reasonably complex with many dependencies and different use case scenarios. Because of that, there is a reasonably high chance that some sort of defect has snuck into an application. That’s where testing steps in.

To quote the famous Dutch computer scientist Dijkstra: Program testing can be used to show the presence of bugs, but never to show their absence!

That being said, the goal of testing is to find as many bugs as can be found.

The most popular mobile operating systems today are iOS and Android, so I’ll try to focus here on how we do testing for these devices.

The Android ecosystem is very diverse and fragmented. This is what the version distribution looks like at the time of writing:

What we can see from this is that Android 2.3.X (codename Gingerbread), still holds the largest market share, although it was released more than 2 years ago! To make things worse, device vendors (Samsung, HTC, etc..) make their own changes and “improvements” to the Android OS which sometimes results in weird bugs and inconsistent behaviour.

To be able to test android applications this is what you need to have:

Of course, even if you run an app over all of these devices, you can never be sure that you picked up all the weird device specific bugs that pop up on Android phones. But that’s the price of having an open (for the most part) operating system.

You can use the Android emulator to aid you in testing. The emulator, however, has two major problems.

The first one is the vendor-specific behavior I mentioned earlier. More often than not, this results in the emulator not being very trustworthy.

The second problem is that the emulator is painfully slow, taking it several minutes to get up and running. This is due to the fact that it emulates an ARM processor so it can run the actual code of your application. It’s also a CPU hog and generally a pain in the ass to work with for developers or testers.

Since only Apple manufactures iOS devices, the number of different iOS devices is much smaller, and the whole ecosystem is a lot less diverse than what is the case with Android.

Taking this into consideration, the iOS version distribution looks a lot saner:

What you can see here is that a major part of the user base is running the latest version of iOS. This is very common behaviour for iOS users – to upgrade fast and often.

Keeping that in mind, to test iOS applications, typically you would need:

When we say “iOS devices” this includes iPhones, iPads, iPad minis and the like.

You can also use the iOS Simulator to aid you in testing your iOS applications.

Notice how Android has an emulator, but iOS has a simulator. There is a subtle, but very important difference. The iOS Simulator does not run the same application code that runs on the device. The code compiled for the iOS Simulator is really x86 code, as opposed to ARM code that runs on iPhones and iPads.

It means that the iOS Simulator starts up in seconds, rather than minutes. It also means that the hardware conditions are not simulated, as is the case with Android. With that said, I have found the iOS Simulator to be accurate enough to use it as a help with testing. It is, however, not accurate enough to replace testing on real devices.

Notice when I talk about testing here, I’m not actually talking about automated testing. We call this sort of testing device-testing, and the goal is to find layout/design bugs, bugs on different operating system versions, bugs on different screen sizes, or with different vendors.

Apart from mobile, we also do a lot of web apps, so in my next article, I’ll write about how we built the test lab for testing web applications and handling different browser versions.