Testing a new mobile app is a process that is made complicated by the endless possibilities of factors that could affect its functionality, including device type, settings, and environment. LYDIN CAMILLERI’s master’s dissertation solves several issues related to this.
A lot has changed since Motorola invented the mobile phone in 1973: they’re more compact, they’re more affordable, and they are used for more than just making calls on-the-go. In fact, the rise of the smartphone in the noughties created a scenario where our devices are basically libraries of applications that can do everything from checking our heartbeat in real-time to scanning documents using our phone camera, and from smoothing out imperfections on our pictures to tracking the menstrual cycle. But not all apps are built the same, and one thing that unites the best one is the rigorous testing process to go through prior to roll-out.
“The testing phase is crucial as it ensures that any bugs [flaws in design] are addressed and fixed so that the app functions in the way it is intended to,” says Lydin, who is a Lead Test Automation Engineer at APS Bank and is currently reading for his Master’s in Computer Science.
“But there is a major issue with this testing cycle, and that’s that mobile app testing is hindered by the Test Scenario Explosion Problem (TSEP). Basically, this describes the fact that there is a practically endless combination of factors that may impact the app’s performance. And I’m not just talking about the type or brand of the mobile phone the app is installed on, but also the version of the operating system; the settings, like the brightness, or whether Bluetooth or aeroplane mode are on or off; and other environmental factors, such as mobile operator, coverage, and sensory input.”
Currently, developers use one or a combination of four testing methods for this purpose: the device-based, where they install the app on an actual device and test it; the virtual-based, through which they test the app via an emulator that mimics certain operating systems; the cloud-based, which utilises the services of a remote laboratory to test the app on multiple devices and operating systems; and the crowd-based, which basically releases the app for users to test it.
The issue with these methods is that each of them comes with some obvious pitfalls. For example, even the most resource-rich company couldn’t afford to buy every device on the market and, even if they could, it would be impossible to anticipate every potential combination of settings. Meanwhile, the virtual testing method means that the app is being tested on just the software, rather than the hardware, and completely ignores environmental factors; the cloud-based comes with a hefty price tag; and the crowd-based means that the app is actually being experienced by users before it is technically ready.
“This reality inspired me to dedicate my master’s dissertation to creating TestMate, a framework which allows developers and testers to distribute and execute automated tests remotely across a pool of community devices,” he continues. “This, in contrast with the other current practices, makes use of real devices that reside in the community and hence do not follow a clinical or lab setup. It also supports developers in generating permutations against device selection and context-related criterion to try to mitigate the TSEP.”
“This real-world device testing process is scalable and opens a whole new world of possibilities for developers”
To get to this phase, Lydin focused on both academia and industry, looking at what the main challenges being faced by developers are. In fact, the way TestMate works is an exercise in reaching a balance between industry- and user-needs: users would sign up to be part of the TestMate community and give a central agency the rights to test apps on their phones in the background. Developers would then submit their testing request and instructions to TestMate, which would then run the tests on their behalf, therefore respecting the rights agreed upon by the user.
At the time of writing, TestMate is only able to run tests on Android phones, mostly due to iOS phones being more restrictive with their security. Even so, this real-world device testing process is scalable and opens a whole new world of possibilities for developers, while solving one of the major issues with device-based testing, which is that companies must constantly upgrade their fleet of phones to keep up with market changes.
From our end, our favourite part of TestMate is that it is the result of a master’s dissertation that unites Lydin’s experience and expertise with something that he is passionate about. In fact, we hope this will inspire others to try their hand at solving issues and drive progress.