The Challenges of Testing an IoT App Using Bluetooth

Internet of Things (IoT) solutions have been springing up like mushrooms after the rain as companies around the world race to bring more products to market as soon as possible. We’ve also been working on one of those solutions in collaboration with Signify, the Philips MasterConnect system.

Supporting Signify’s smart lighting system from the software side, we’ve developed mobile applications for Android and iOS, and performed quality assurance for both.

Working with IoT apps, testing can bring a specific set of challenges. Additionally, since Signify’s apps use Bluetooth as a core functionality, there are many different use cases and devices the applications need to interact with.

This large-scale IoT project brought us face-to-face with many challenges. Here’s an account of how we approached and overcame them.

Working on any IoT project, especially a complex one like the Philips MasterConnect system, the first challenge is getting to know it. To be able to test something thoroughly, we first need to have a deep understanding of how it’s supposed to work.

In our case, the system consists of various types of lights (usually packed into in-luminaire nodes so that the smart sensor doesn’t show), wireless Zigbee Green Power (ZGP) switches, and ZGP sensors. To set up a network, the MasterConnect app:

The process is then repeated for each additional light added to the system. Once complete, the user can configure light behavior according to their specific needs.

Users can also expand the network’s complexity by adding ZGP switches and daylight or occupancy sensors. To do that, the MasterConnect app:

By doing this, the user can have additional manual control of the lights or add another layer of smart automatic control.

As you may have noticed, the system has several complex layers that we testers need to know and understand. We are expected to grasp how Zigbee works and know our way around the multitude of different supported products and their respective firmware capabilities.

Meanwhile, the user is completely unaware of the complexity of the process as it all happens under the hood.

If we want to handle this successfully, the best approach is to break it down to basics and focus on understanding each part of the system. For this case, we prepared an in-depth 100-page-long Project Handbook with all the nitty-gritty details we had to know. It added quite some time to the onboarding process but produced a better outcome in the end. We were able to detect possible problems early on, as early as the architecture phase.

It’s precisely for these reasons we have testers involved in each architecture-defining phase, both for mobile app implementation and on a system level. So the lesson here is to get down to the very basics of system components and get involved in the development process as early as possible.

Testing mobile apps that use BLE as a core functionality means you have to take into account two types of hardware:

For mobile phones, this includes both models and OS versions. You should also understand that some devices may look the same on paper, but their reliability might differ substantially in practice.

In our case, this was especially true for Android devices. We discovered early on that most Huawei mid-range or low-budget models can be very unreliable when establishing and maintaining a BLE connection. Compared to similar Samsung or Xiaomi models, Huawei devices consistently required us to restart their Bluetooth module by disabling/enabling it in the phone’s settings or reboot the entire phone. That’s also the reason why those devices are no longer supported.

However, you have to be careful to avoid the trap of testing only on top-of-the-line high-end models like the Samsung Ultra edition.

Another notable example is the BLE over-the-air (OTA) upgrade. We conducted iOS testing on an iPhone Pro that seemed fast and reliable. However, when we re-tested using an older iPhone like the iPhone SE – the thing broke down. To address this, we adjusted the implementation to work seamlessly and without interruption on any supported iPhone,  whether it’s the latest Pro Max edition or your trusted old iPhone X.

As mentioned before, knowing the system means you have to understand the expected use cases, which devices the mobile app is expected to connect to, and what those devices’ capabilities are.

The MasterConnect solution brought to market several different types of lights, ZGP switches, and sensors, all of which rely on the Zigbee mesh network for communicating with each other. However, different devices have different capabilities, and they all need to work together to bring the expected system performance to the user. There were several subsets of challenges to overcome here:

Another challenge I would like to address is the so-called shift-left approach. In a nutshell, it’s about starting testing as early as possible, and in our case, we started in the architectural phase, as mentioned before.

We need a high-level understanding of feature requirements since we often work closely with the design team. Sometimes we accompany them for usability testing or review the data obtained there in collaboration. In many cases, this allows us to be the link between the architects designing the feature and the UI/UX designers who are working on preparing the UI proposal.

Usability testing data allows us to put ourselves in users’ shoes while testing Figma prototypes helps the entire team address potential issues or misunderstandings before they even arise. What this means in practice is that we don’t simply implement features on request but rather challenge certain aspects and provide constructive arguments about why something needs to change.

Treading the fine line between maximum quality and delivering a feature to market quickly is no easy task. However, by getting involved in a project early on, we gain the knowledge and experience that helps us tread that line more easily.

That being said, proposing a coherent and workable strategy for a project involving multiple software testers and test automation engineers can be challenging.

Since the entire team works in an Agile setup using SAFe methodology, we usually prepare a high-level testing strategy every three months. First, we define the features we will be working on and the number of releases we’ll have. The testing strategy is then broken down into smaller chunks using JIRA’s Xray test management tool, where we prepare our test plan and multiple test executions.

The sheer complexity of a large-scale IoT project will often bring about a large number of complex test cases. This in turn results in regression testing (and the whole release process) taking longer and longer with each new functionality being added.

Since the modern market is very dynamic, you always want to reduce the time needed for releasing new functionality while ensuring that the product quality remains consistent or increases.

A possible route for achieving this is test automation, and one of the biggest challenges of working on a project involving Bluetooth is figuring out how and which test cases to automate.

Let’s start with how. Since we are doing product-level testing, we have to make sure the mobile app is working according to the system and product architects’ requirements. 

After careful consideration, we concluded that the best way to go about this was to create a mock environment (server). We would intercept the Bluetooth traffic there before it reached the actual Bluetooth layer on the mobile phone and returned the smart devices’ expected responses. This way, the developers could write down integration tests while we were able to write end-to-end (E2E) tests mimicking the actual hardware, thus protecting the real devices from possible unexpected behavior (negative scenarios were being tested separately). 

To achieve this, we had to work closely with the developers to make sure all the features were supported on the mock server. The test cases were first executed manually on real hardware to ensure the implementation was correct. Once confirmed, that test case was automated, so there was no need to execute it manually during the next regression test cycle. This way, we were able to reduce the time needed for completing regression testing and lower the release cycle.

When choosing which test cases to automate, we had to rely on the mock server’s current maturity. We started with UI-related test cases as they proved to be easier to automate, which allowed us to tackle more complex BLE-related scenarios and research in the manual part of testing.

As the complexity of the mock server improved, we started to automate more and more complex BLE-related test cases, so we could start testing new features instead of spending too much time on regression testing. 

However, those tests rely solely on the mock server’s capabilities, so we had to find a delicate balance between enabling support for existing features and developing new ones. To help us determine which test cases to focus on, we took the data from previous regression test runs and extrapolated the most critical ones. Those critical test cases were prioritized and automated first while we dealt with the rest as time allowed. 

When most of our backlog of critical test cases was automated and done, we shifted our focus to automating test cases that covered new features. This way, we ensured the end product was sufficiently covered with both manual and automated test cases.

Working on an IoT project brings a special set of challenges and complexities. It is important to take the time to familiarize yourself with the intricacies of such a project and the needs of the end user. This requires close collaboration with development and design team members. 

When testing on different mobile devices, you should make sure not to rely solely on the advertised hardware and software capabilities and instead test on multiple devices while taking note of their performance. Use that knowledge to devise a proper testing strategy. Revisit it often and adapt it when needed. Try to automate what you can so you can free up time for focusing on more complex challenges.

Or, in the words of experts: