Can You Hear the Bug? – An Intro to Audio Software Testing

Software testers are usually versatile as they need to be able to work on digital products for any industry. However, music testing comes with its own set of challenges best tackled by QA engineers specifically attuned to audio issues.

People in the tech industry know all about software testing and software testers, but audio software testers are not that common knowledge. At Infinum, we’ve worked on a number of projects in the music industry and got a close-up view of this highly specialized field. 

Who is an audio software tester and what is it that they do? In this article I’m going to give a brief outline of the role, some of the challenges of the job, and some of the methods we use to resolve them. We’ll look at some similarities and differences between audio testing and software testing more generally, and some useful pre-requisites for the role.

A software tester, or a QA engineer, is a person with a wide range of skills, and knowledge in a variety of different fields – a jack-of-all-trades, you could say. That does not mean that a software tester who is very skilled in one field is bad at testing, but most of the time we test a lot of different apps that require a lot of different skills. Variety is the spice of life, right?

On the other hand, a specialized tester can dive deep into a field and get familiar with all its intricacies. It turns out that audio testing is so specific that there are companies that specialize in it. These Audio QA laboratories perform both manual and automation testing and have organized their entire setup to accommodate this niche. 

In any case, when testing music apps, you do need an audio-specific testing perspective – and this is where audio testers come in.

Audio software testers are mostly required in the gaming and the music industry, but also in video testing and hardware testing. For example, testing music equipment, headphones, and so on.

Most job descriptions for audio software testers include requirements like “strong ear for sound anomalies and the validation of sound quality.”

This is not to mention how-to-connect videos, when you need to test hardware such as MIDI controllers or other instruments and equipment. 

Aside from music theory, an audio production glossary and other musical experience can also come in handy. You don’t have to be an accomplished audio producer and know the ins and outs of recording studios to test a MIDI controller. The only real constraint is that you cannot be tone-deaf.

When it comes to testing audio, things can get very complicated very quickly. Yes, we all had physics class and slept through the wave and sound lectures, but audio testing is not just waves and decibel counts. If you are a tester who has found themselves on a music project, but don’t know the basics of music theory, there are a few options you can explore (provided you have no innate hearing barriers).

The last option can be problematic because you’ll have to contact the client a lot, and it’s not their job to test the product and explain the basics – that’s why they hired a QA person. The best and cheapest option is therefore to have a specialist audio software tester who’s familiar with testing, music theory, and/or audio production. 

Without this specialist knowledge, a tester will probably miss some audio-related bugs, which will then go to production, and the only way to catch these problems will be through angry users, or worse, clients. 

Another great way to get valuable feedback is to work with real users and check with them what their needs and habits are when using such an app.

Now that we have some sense of the skills an audio software tester needs to have to test music apps, let’s discuss some audio bugs. When it comes to testing audio features, your ears are the main tool, and as with any other tool, you need to be able to rely on them – hence you should have a good ear or two.

Sometimes, even a small alteration like changing the tempo of the song can make the audio misbehave. For example, I experienced a bug in one app where the piano and bass would go off-key when the user changed tempo, but only during the very low notes in the song. It was very noticeable when you changed to a lower key. 

Another example was when pausing a song on a MIDI controller and playing it again, the dynamic click would go out of sync with the song. The dynamic click ended up being very challenging because I could break it with various different steps, one being changing the BPM multiple times. Another way to get it out of sync was to play the song three times in a row from the beginning to the end. Not only did I have to listen to it multiple times, retrace my steps and then reproduce them, but I had to be creative too, and keep finding new ways to make the metronome go out of sync.

And hey, what is an app without a crash? Some crashes were only reproducible on the latest OS versions, and one of them would happen when the user changed the key and the BPM, and when they clicked play or pause. You should note that another problematic feature which can often produce crashes is transitions, where one song switches to another while using an audio effect set by the user.

Let’s say that you have been testing clicks all day, and by the end of it you barely know your name, let alone if the downbeat falls on the right beat of the song. What do you do then? You need to have a sanity check with your team, of course. But, what to do when you need a sanity check from your teammates but they cannot hear what you hear? When I was in this situation, I recorded a part of the song with everything working as expected and then reproduced the issue on that same part for comparison. If people are not immersed in audio testing, it helps to have everything prepared and point out what exactly they should focus on.

Another important area of testing is, of course, UI/UX testing. For example, during the development of one app, I started to realize that the main play/pause/stop/next and previous buttons were too small while the song was playing. This became even more prominent after adding other features and more buttons. When I tried to click the next button, I would find myself hitting the stop button, or repeat section button.

When you have hardware that you need to connect to your device, how-to videos become your new best friend, as well as user experiences and forums. At first the hardware may look scary (even if you’ve used MIDI keyboards before), and you start to whisper under your breath: “How am I going to test this?” But once you get the hang of it, it is not that scary or hard. MIDI controllers are just a drop in the ocean of other, more complicated hardware, in which you need to learn to swim.

Testing a music app manually, especially its audio features, can be extremely exhausting. Sometimes when you have been testing clicks all day, at some point, everything starts to sound the same. Since humans are not robots (yet), there are three approaches to consider. 

The first is to stop testing, and to get some rest or even leave the issue for a day or two so that you can come back to it with a fresh perspective. The second is to try to automate some processes. The third option is pair testing with your teammates, because they can give you a fresh perspective and will potentially discover problems that would otherwise pass under the radar.

Creating and maintaining test cases is very important as the development progresses and new features are released, and a tool that helps with it can be very useful in any project.

We can get a good overview of an app’s performance after we’ve done a test run, and it helps a lot when it comes to regression testing. Regression testing is very important and needs to be executed before the release of any new feature. It can often be the case that when one bug is fixed, two others appear in its place. Since music apps often have a lot of additional hardware and are cross-platformed, regression testing is a must.

As an audio software tester, or any other software tester, you always have to keep in mind that you need to try and use an app like an end-user, but also to think out of the box, to catch as many bugs as you can before the production release. 

What separates an audio software tester from the rest is the ability to hear music, or in this case, mistakes in the code.

It is important to remember that it is not enough to rely on hearing alone, and that you don’t have to have a perfect pitch to test a music app, but that you do need to be creative and always think of new ways to find bugs. 

But most importantly, you should enjoy the overall process of software testing, audio testing, and persistently finding and solving software and audio bugs.