A good mobile test strategy requires enterprise architects put in place a comprehensive test automation framework....
There are some upfront costs in implementing this in terms of time and infrastructure. However, this will lead to higher-quality apps that are more aligned with business goals. This can augment a DevOps strategy that enabled faster time to market and a quicker pace for iterating new changes.
"The DevOps principles of high levels of automation and communication within the organization can certainly aid in testing mobile apps with greater success," said John Busch, CTO of Moasis, based in Austin, Texas. "Ensuring a tight SDLC [software development lifecycle] is in place with automated CI [continuous integration] that runs unit, regression and UI tests will provide the infrastructure to be successful in such an endeavor." It is also important to prioritize intelligent testing and incorporate security testing into the test infrastructure.
Prioritize intelligence over mindless automation
The very nature of the fragmented mobile ecosystem and the need to support different types of users can be challenging. Automation can help speed the process, but it can only go so far on a limited testing budget. A good practice is to prioritize what to test first, said Mush Honda, quality assurance director at KMS Technology, an IT service provider based in Atlanta. "Start by focusing the testing approach on the features most important to your end user," he said.
This requires a thoughtful, almost experimental approach to testing. Honda recommended organizations encourage testing teams to use exploratory testing processes that empower testers to find the edge cases that are likely to cause problems. This makes it easier to test features the enterprise wants and needs first, so developers can resolve issues faster. Exploratory testing also allows the team to run an automated testing suite in parallel to save time.
It is also important to use a pragmatic testing strategy. Honda recommended using an optimal mix of devices in testing that vary by location, type of device, OS version, configuration and connectivity options.
Start with a test automation framework
A test automation framework can make it easier to spot issues and facilitate useful communication about what caused the problem. These can also be incorporated into developer tool sets, so developers can fix most problems before the code is passed onto the testing team.
Enterprises also stand to benefit by adopting a mobile-agnostic framework, said Jon Thomas, principal product manager at BMC Software Inc., based in Houston. "Even if you are 100% sure that you will only be supporting iOS or only be supporting Android, you might want to consider selecting an automation framework that works for both operating systems." This will allow the enterprise to reach out to a more diverse audience in the long run, without retraining developers and testers. Appium and Calabash are both good open source options that support iOS and Android.
Next, the enterprise needs to decide where to test applications. Device emulators and simulators may be good options for quick development tests, but it is important to use real devices for acceptance testing. One approach is to build and maintain a private cloud of mobile test devices. "However, unless you have specific requirements to keep the applications and tests on premises, you will likely want to leverage a public cloud service like AWS [Amazon Web Services] Device Farm," Thomas said.
Security testing in your mobile test strategy
The testing strategy should also include security testing as part of the process, said Carsten Eilers, a security consultant in Germany. Fixing a security flaw during development is much less expensive than fixing it in production. This also reduces the risk of consumer or regulatory wrath if a vulnerability is discovered in the wild.
Mobile security also needs to be tested across a wide range of devices and OS versions. Eilers said it is especially important to do this testing with rooted and jailbroken devices. "A really secure app is not only secure if it runs on a secure system and a secure device, but also on an insecure system and a manipulated device. It's impossible to detect the rooting or jailbreak if someone wishes to hide this fact. That's in the nature of such a manipulation."
A good practice is to embed security people into new projects, along with operations and design experts. "If you have continuous changes on the code and continuous deployment of the software, you also need continuous security tests," Eilers said.
Security testing is hard to reliably automate
A number of security tools can help to complement a DevOps workflow for mobile apps. Two common classes of these tools include static application security testing (SAST) and dynamic application security testing (DAST). SAST is faster because it just analyzes the code to find blatant vulnerabilities. DAST looks for vulnerabilities caused by the apps' behavior. DAST tends to be more comprehensive and also more time-consuming.
"It is important to have mobile applications tested with manual DAST owing to the lack of automation technology in mobile application review," said Shreeraj Shah, research head at Blueinfy, a security testing consultancy based in India. Some of the key vulnerabilities to look for include sensitive information stored in local storage, sensitive information stored in shared locations -- keychain or shared preferences -- and sensitive information sent to third-party servers.
Other types of checks, like permission and API usage, can be covered more efficiently with a SAST methodology. "DAST helps in simulating actual traffic and capturing local storage usage with higher confidence compared to SAST," Shah said. "Also, DAST enables a quick turnaround, which is key for mobile due to the high frequency of releases."
Some security testing tools are starting to emerge, but the field is still immature, Shah said. Many of these tools end up generating false negative and false positive results, which need to be validated quickly as part of the development lifecycle. If an enterprise is overly enthusiastic about automating security testing, it is easy to end up with a sense of security, rather than real security, Shah said.
Apply a system-wide focus
A good mobile test strategy needs to align with the overall goals of the business. Simple functional testing will only go so far in meeting these needs. Continuous integration, continuous testing and continuous delivery are all practices in an overall DevOps strategy. "It's important to look at the goals from a system-wide objective instead of focusing on local optimization within a single practice," BMC's Thomas said.
For most organizations, the system-wide goal is to quickly release compelling and quality applications to their consumers. "If the focus is only on the testing, you can create a local optimization that fails to impact the system-wide objective," Thomas said. The enterprise might create very good tests, but if there are long wait times for a person to manually run the tests, the SDLC can lose speed.
Additionally, if tests are not consistently run, quality can be lost. "By integrating automated mobile testing into your continuous delivery pipeline, you can speed up your releases while still consistently ensuring quality," Thomas said.
Steps for the right testing automation
Using Agile test automation principles
Learning more about the eggPlant testing tool