Testing - Various

Software and cathedrals are much the same: first we build them, then we pray.

Summary

This potentially infinite list is just some minor testing tidbits we gathered through the years that might help you find some sweet, sweet bugs.

Android vs iOS

Comparative testing is allways a good thing to do, it could illuminate a lot of issues quickly, and will help to compare implementations on Android and iOS directly.

Cert pinning

The app uses cert pinning? Try a man-in-the-middle attack with Charles and its root cert. No responses coming through? Good, the devs did their job.

Compare to design

Whenever testing a new feature, make sure to compare it to the agreed upon design (most often found on Zeplin or on Invision).

Pay close attention to:

Could your grandmother use this?

Not sure that all the UX nitty-gritty has been worked out ok? Try stopping a neutral person for a bit of hallway testing and have them try the feature out.

Three to five people are the magic number for usability testing anyway.

Empty states

All screens that display dynamic data (e.g. lists, results, profiles, etc.) should have an "empty state" if there is no data to display. Preferably, the empty state of the screen should also include a call to action along with some persuasive copy.

Get a crowd to help

You've got a lot to test and not enough time? Ask the entire team to test by creating a test plan and organizing a testing session. An example of a test plan for such a session can be found here.

Get a designer to help

When testing a design-heavy feature, go to your designer and go through the look and feel together. They will usually catch some issues that have not been implemented as designed.

Get a tester to help

Try pair testing and invite a colleague to help you out. Double the eyes, double the fun, double the success.

Get off the beaten path

Don't just follow what the task says. Do whatever you think a user might do. Do what a user might not do as well.

Hanging calls

Whenever a network request is being executed (which is often indicated with a loader), try navigating away from that screen. If the developer has not handled this case, it might result in a crash because the app will try to select or add data to screen elements that are no longer allocated in memory.

Heuristics in general

A heuristic, is any approach to problem solving, learning, or discovery that employs a practical method not guaranteed to be optimal or perfect, but sufficient for the immediate goals. Read more about it here.

Locale

The German strings never fit. Make sure to do a smoke test in all supported languages. Other than strings, differing formats may also break certain features, e.g. dates, currencies, time, etc.

Long lists

Scroll down through several pages of a list. Enter the item and change it. Now go back. Is the list appropriately scrolled and updated? Good.

Lose your finger

The app uses fingerprint login? Remove all stored fingerprints from the device.

Monkey testing

Behave like a monkey, try tapping that submit button two or three times. Try tapping navigation buttons several times. The Monkey testing chapter might also help.

No internet connection

Try going through the app without an internet connection. Better yet, try doing an action that will result in a network call and lose the connection in the middle of the process.

Orientation

If the app supports both orientations, make sure to test all screens in both orientations. Try rotating the screen between transitions and see if anything fails.

Permissions? Just say no!

Whenever an app asks for permissions, always say no. Even better, say yes and then say no. What happens?

Prioritize testing

When doing smoke testing, test the major and critical features first because that's where the risky bugs usually hide.

Security smoke test

Run an automatic security test using MobSF or QARK for mobile or OWASP ZAP for web.

See this chapter for more detail.

Sessions

Log in and put the app into the background. Come back after the token expires.

Several screen sizes / aspect ratios

A good habit to get into is testing the app on several screen sizes. Some common aspect ratios and resolutions can be found on DeviceAtlas.

Slow internet connection

Try accessing screens with a very slow internet connection. See if the calls time out, and if they do not, can the user navigate away from screens.

Submitting a form with missing data

If you have to enter a username and password, enter just a username and submit. Nothing happened? That's bad UX. You should know what's missing.

Testing on a slow connection

You can use our own slow network in the office:

On iOS, you can also enable network throttling in Developer settings on the device itself and skip our Kornjaca.

You can use Charles or Proxyman to the same effect on both platforms.

Testing purchases

Some tips for testing:

  1. Make sure that you test both good and bad information.
  2. If you are testing on production(please don't do this) communicate with the client so that they know that some "fake" purchases are going to be coming through.
  3. If the app uses services such as Ayden or WebPay or similar, try using a credit card that has a form of 2-part authentication like a bank token.
  4. Cash on delivery is rather hard to test. You can test in tandem with the client when a user chooses this option. Cash on delivery does not go through a service, rather the payment is collected when a user recives the ordered item.
  5. Check what happens if you lose internet connection during payment process.
  6. Check how the process acts on a slow connection.
  7. What happens if you get a pop-up.
  8. What happens if you close the app in the middle of the payment process.
  9. Verify that you recive a receipt for the purchase, otherwise the payment should not go through.
  10. If the app has a shopping cart, make sure to verify that changing items within it, does not negatively affect the payment process.
  11. If there is tax calculation within the payment process, check how it is calculated and if its applied well.

Most important thing here is to follow the client requests, because the client is going to end up with that payment process and is going to have to work with it. Think of the users :)

The floating keyboard

On an input field, get the keyboard up. Now go to the previous screen. Still up? Well that's a bug.

Try some crazy strings

See the Big list of naughty strings and the Bug Magnet.

Try various unexpected inputs

What to watch out for when testing social login

A lot of the users today like having only one account for all the web places they visit, so more and more websites add the option to log in through things like Facebook, Google, Twitter and many more. The login testing by itself is no easy task, and the social media part just adds more complexity to it. Some of the stuff to watch out for is listed here:


dilbert-god.jpg