How should you approach testing on mobile devices?

Zoonou has one of the largest real device labs in the country, operated by teams of qualified testers. But when would you use real devices to test your applications and websites?

Largely, that depends on factors such as the product itself, the development environment it will be built in, whether it will be a native, web or hybrid app, the ongoing development workflow and the available budget for the project.

Emulators and Developer Tools

If the build is going to be a native iOS or Android app then you will probably make use, in the early development stages, of iOS Simulator or Android Studio. These proprietary tools allow developers to quickly progress their work, checking the code and architecture to ensure the product is emerging as expected. The limitations of these tools are that they are software packages designed to inspect the progress of other pieces of software.  They don’t replicate the experience of a genuine user and do not address the performance of an application when accessed on a specific piece of hardware.

This is also true of non-native app types or websites where the choice of development environment (IDE) will usually determine the tools that are used to create and check the progress of written code. Chrome is the dominant desktop browser and its Chrome Developer Tools (DevTools) is, not surprisingly, one of the most used for its suite of diagnostic and debugging capabilities.  It also features Device Mode that will simulate a range of mobile devices, allowing developers to get sight of their product in a software environment.

From this point forwards in the development cycle any test choices should include real devices, as this warning on a DevTools page tells us.

Part of what informs the next testing option is about how sharply the end user needs to come into focus. This is generally a risk calculation taking into account the profile of the product, who its owner is, the structure of the project management team and to some extent the culture of quality assurance that exists within an organisation versus expediency.

Device Clouds

Device clouds are facilities that allow remote access to physical devices. Device-cloud services use VNC (Virtual Network Computing) to transmit a user’s keyboard and mouse events to the device the user is connected to.

Here the test options increase as more of the real-world functions of the device become available. However, while device clouds utilise real devices – the nature of the remote connection means that testers do not interact with the handsets as real users would. Scrolling, tapping, pinch-to-zoom, are all replicated using a mouse, meaning touch points and clickable areas are not tested in a way that is representative of end users. Any applications that require hardware integration, such as the device camera, microphone or audio can also be problematic and limiting.

One of the assumptions associated with these facilities however is that a test approach will be employed.  This can either be handled by internal test specialists or by engaging an independent test company to write scripts and conduct testing.  Often though, a compromise is made by using available team members who are not necessarily test experts who may also need time to familiarise themselves with the platform. Test outcomes will obviously vary depending on which method is employed.

Security Concerns

Another consideration for device cloud users is the way security is handled.  This is specific to the device cloud. For example, some use a proprietary process that wipes every device at the end of the test session, others however, restore devices to factory settings. Additionally, the operational location may affect security policies and procedures and product compliance issues may be affected.

Occasionally, there may be the need to do testing on sensitive projects. This could involve local testing, where IPs need to be whitelisted so that the website can be accessed on the devices required. The testing of a website like this usually isn’t possible when using cloud-based services. However, some services allow traffic to be rerouted through a local connection (slowing down network performance of the test, but allowing the user to test something they would not be able to otherwise).

Testing can be further complicated when it involves installing the application under test (AUT) using a beta app launcher. The AUT is usually restricted to the devices it can be installed on, using the device IMEI or UDID as a key.

If a device without an IMEI in the whitelist for the AUT attempts to download the App, security measures prevent this from happening. Using a non-remote, physical device testing service, a client can be provided with a single IMEI or UDID for each device. Once whitelisted, the application can then be downloaded and tested on the physical device.

When using cloud-based services, there is no guarantee that the device that has been accessed is the same physical device between test sessions, so multiple different IMEIs will need to be provided for whitelisting. Additionally, as the devices on the cloud are wiped clean between sessions, the need to re-install the AUT per test session adds time to the test effort.

The above issue does not occur when testing an application in-house, as beta application launchers do not need to be provided to install the application. Instead, direct access to the APK (Android) or .ipa (iOS) can be provided, behind some level of security, reducing the complexity required to get the AUT onto the device being tested.

Automation

Implementing test automation is dependent on the device cloud that is being accessed.  Some allow the testing of native iOS and Android apps using Calabash for example, others allow the use of Selenium to automate tests on browser based apps. Some offer more basic record and playback automation.

Whether using device clouds or not, automating tests on devices requires the usual considerations such as who will write the test framework, is scripted manual testing necessary first, and how will the automation effort be maintained? Such questions inevitably require input from a specialist and the necessary budget to address them.

Real Devices. Real Environments. Real Users.

The other part of this test landscape is the obvious one of using real test analysts, testing on real devices. In most cases the devices used will be part of a service package and not charged at a separate rate.  Most importantly, products will benefit from getting tested on environments that match those of the actual end users.  How does the app or site perform, react and feel when accessed and interacted with on the actual device it’s intended for?  It’s a scenario that can’t be replicated in any of the other testing methods.

Too often, real device testing is seen as a User Acceptance Testing (UAT) function at the end of a product or feature release cycle and not part of an overall quality assurance process.

Real device and user testing should not be seen as a UAT luxury but as part of a quality assurance commitment.  Of course this approach could be used for most of the development and release cycle but the reality is that it is usually introduced as a blend of test methods.  Having an independent ‘test partner’ that will take the time to understand both the development and product owner needs will help to determine when it makes sense to use the right test method. This extends to regression testing and the maintenance of any automation that may be needed.

The key challenge of real device testing is maintaining and updating a large range of mobiles and tablets – which can be expensive and time consuming. That’s where an independent test company can really be of value, making it a dedicated priority to regularly update their device range with the latest hardware and software, while maintaining a collection of older devices and OS versions.

The Future: Augmented and Virtual Reality

As with all fields of software development, the testing landscape will change. Emulator and device cloud technology will of course improve and adapt. But as we see the emergence and growth of interactive user experiences such as Augmented Reality (AR) and Virtual Reality (VR), the need for real testers and real device testing is more pertinent than ever.  These interactions require testers to manually engage with devices in real space and time, and factors such environmental conditions, device performance and overheating are pushed to the forefront.

Zoonou has been helping developers and business owners make the right decisions about how they test on devices since 2007. Part of the reason why clients bring Zoonou in on over 300 projects a year is because we are able to give advice on a test approach based on experience that ultimately saves them money and delivers great products.  Using a blended test approach including test analysts interacting with real devices will ensure that the stage, product and budget are all factored into the test effort.

To find out more about how Zoonou can help to advise on your device testing strategy, please see our QA Advisory and Consultancy page. To request a list of all the mobile and tablet devices we have here in our test lab, please see our Device Lab page. For more details on our compatibility testing service please contact us at info@zoonou.com and one of our team will get back to you. If you’d like to get in touch about anything else, please head over to our Contact page.

Share this post