10 Principles of Successful App Testing

I guess there are two kinds of people in the world: the ones who make things and the ones who break things. Until now, The Capsized Eight mostly featured stories by people from the first category. Not this time around. This story is all about the ways to put those painstakingly coded apps to the test. Mostly for the benefit of the final product but sometimes for the pure joy of seeing them capsize.

Why would we do that? Well, mobile apps are an unusual beast. They are supposed to be intuitive in a cramped space, fast on limited hardware, look good on a bunch of different devices and stand out against the extreme competition. One of the things that may help you out of these contradictions is a relentless quality assurance procedure.

Often there will be dozens of variables you’ll have to keep in mind when testing mobile apps. After a while they might become second nature but a bit of revision never did any harm. In light of that, we’ve compiled a list of prescriptions and tricks every tester should have up their sleeve. Keep in mind, this is by no means an exhaustive list but a broad overview of approaches and good practices that will help you navigate the labyrinth of testing mobile applications.

Let’s paraphrase Fight Club’s Tyler Durden and make one thing clear as day – the two most important rules when testing mobile apps are:

We’re stressing this because your apps will not be used on emulators, users do not care about emulators and emulators do not represent the real state of affairs. The only time an emulator should be used is during development. Even though this might seem self-evident, we’re not completely sure people still take this point as seriously as they should.

The next three rules continue the trend:

Ideally, you’d test your app on every single device out there. Since this is impossible, especially in the case of heterogeneous ecosystems such as Android, you’ll have to get smart about it, prioritize, and cover the largest possible chunk of the market. You want your application to look good, feel good and perform smoothly on a large sample. It’s not all about the current flagships.

You’ll want to check for interface consistency and general backward incompatibility. What works on Android 6.0 doesn’t necessarily work on 4.1, and if 4.1 is supported, speak to your nearest developer.

If you’re testing version v2.2-42 of your app and the entire changelog is “fixed on/off toggle for muting”, there’s no point in wasting 10 hours going through each and every use case scenario involving things that have nothing to do with muting. If your developers are used to deploying beta builds on a regular basis, your job will be checking bug fixes (resolution testing) and additions. You can do a quick run through the fundamental features of the app to see if anything else is affected (smoke testing) but leave the rest for later.

However, if you’re testing version 3.0 which has a changelog the size of your leg, a passing glance probably won’t cut it. Whether you tested each step of the way or not, you should probably do a full (regression testing). New features should integrate smoothly with old ones, nasty bugs of yesteryear should stay out of sight and the app should behave as a polished whole. Go the whole nine yards: take the time and effort to bite it from each and every angle. Discard the fact that your developer might exhibit a serious case of hate towards you due to those bug reports that keep flying in. More bugs reported by the quality assurance team means fewer bugs reported by the end-user – and this makes everyone happy.

Is your app used for finding and buying tickets for jazz concerts? Then the user should be able to find and buy tickets for jazz concerts. However simple or complex, the app should do what it’s meant to do day in and day out.

When doing (functional testing) you should check whether the app succeeds in the aforementioned and think of possible scenarios in which a future user might stray off the beaten path and destroy those functionalities and your developers’ weekend.

When testing a complex application, going through a set of dilligently written use cases is almost always a good practice. Instead of relearning the app every time, simply consult your documentation and update it when needed. Your future self will applaud you, as will any other tester or project manager that tries to make sense of all those features.

Usability testing is another cornerstone of mobile app testing and it is concerned with those parameters that are difficult to measure or categorize. Mobile devices are small and have very few physical buttons, so apps require extra effort to become pleasant and usable. Ask yourself the following questions while testing the app:

These might sound abstract or banal, but you should be trying to capture some general impressions. Your app might look good on paper and do what says on the box, but if you’re noticing it has a low retention rate, it might suffer from a nasty case of terrible user experience. Other than your insight, use Google Analytics and its detailed data when determining where things may be going south for your app.

Lend a helping hand if you have spare time and possess the know-how, but your priority should never be debugging other people’s code, pen testing or doing too much API traffic sniffing. You should convey what went wrong and how it went wrong. It’s often best to leave the why for those who have a broader understanding of the underlying logic. Treat your app as a black box and stay as neutral as possible.

Having said that, always describe issues into excruciating detail. Report the exact app version, mobile device, operating system and steps to reproduce with each and every bug. If your bug report is nothing but “crashes on the first screen”, you are helping no one. Also, Crashlytics or a similar crash reporting service should make things easier when trying to single out the exact point of failure.

The devil is in the details, and 9 times out of 10 rushing will get you nowhere. There are several approaches to testing, but none of them include being lazy about it. Here are a few questions you should constantly be asking yourself:

Sometimes you might overestimate your familiarity with the app and assume something works. Never assume it works, always assume it is broken.

Wanna know how resilient your app is? Do some destructive testing. Try going crazy and tapping all over the place. Better yet, let The Monkey have a go while you’re contemplating your next move. No crashes or freezes? Get your developers a refreshing beverage; they deserve it.

When system permissions are concerned: just say no. Is the app asking you to accept a camera permission on your minty test device? Always reject the developer’s attempts to woo you. 7 times out of 10, they will have forgotten to account for those that can’t be wooed easily which might lead to crashes, broken functionality and general awkwardness.

Is your app already used in production? Well, contrary to popular belief, the common user won’t uninstall the app, clear out the cache, update and reset their OS prior to updating it. Download the newest build and update the messiest, overloaded instance of the app you can possibly conjure. And just for fun, let it be on a device with the minimum supported OS version. If it starts crashing, don’t even blink – just report.

Most applications today will be communicating to a server and thus expect some sort of internet connection. This is crucial and needs to be tested thoroughly. An average user will use your app on networks ranging from splendidly fast Wi-Fi to glacially slow EDGE (named so because it gets you to the very edge). Some might start the app offline; some might traipse into a tunnel or a basement and lose their connectivity; others might be plain stubborn and keep connecting and disconnecting from the internet. Regardless, your app needs to work in all cases. If there’s no connection, the user should be aware of the problem. There should be no crashes and no loading animations going on forever.

If your application expects Bluetooth or GPS to work – validate that it also works without it. A message should be clearly communicated to the user: “Hey man, please turn on this or that in order for this feature to work.”

Lastly, if you chuck your app into the background and continue using it a couple of hours later – please make sure that it doesn’t crash on wake-up or drain your battery life in the meantime. A power-thirsty app plagued with memory leakage is bound to end up uninstalled.

Introduce continuous testing under the radar and let your scripts do the talking on a couple of test devices you have lying around the office. Take advantage of those use case scenarios you’ve diligently written down and code or record a couple of useful scripts that’ll spare you from testing those most banal and repetitive parts of the app. If you’re feeling especially enthusiastic, you can even try running your script on a few dozen devices by using a device farm such as Amazon’s.

Calabash, Robotium, Appium, Cucumber, Espresso and MonkeyTalk are a couple of frameworks worth considering. If you’re testing multi-platform apps, Calabash (Ruby only) and Appium (Ruby, Java, JavaScript, C#, PHP) might be your best bets.

I’ll let you explore these on your own time for now since going through the strengths and weaknesses of each would be a blog post in its own right.

If you get fed up or feel like you’re not seeing the app clearly (tester’s fatigue), ask a buddy to have a quick look. Looking at something for too long and at an extremely close distance narrows your understanding and critical faculties. If you can’t see the forest for the trees, a fresh pair of eyes will almost always help.

In the end, all we want to achieve is helping build apps which people will enjoy using and keep as a part of their daily life. There are many more things we could have mentioned, but I guess everyone has their own top 10 when doing mobile application testing.

One thing is certain: testing cannot be circumvented. Try downloading a couple of top apps from Apple’s App Store or Google’s Play Store. Are they crashing? How often? If they aren’t crashing at all – the dev & QA teams have probably worked in unison to achieve that. Crashing so often you want to pull your teeth out? A tester must be sipping cocktails on a beach somewhere not doing their job. My humble guess would be that most belong to the first category. So don’t slack when testing is concerned, your business will thank you.