Why Mobile App Testing is So Hard

Not that you need reminding, but mobile app testing is difficult. Beyond the fact that it’s still a fairly new medium and testers are still figuring out the nuances of mobile app testing, the single biggest challenge is the sheer size of the mobile ecosystem.

Even when focusing on a single operating system, the number of different devices, screen sizes and OS versions – not to mention carriers and locations – make testing a single app quite complicated. To illustrate this, let’s take a look at this recent Engadget article discussing the launch of the BBCSport app for Android. According to the article, the BBC’s newest app “is compatible with Android devices running version 2.2 or above and while the Beeb has worked to ensure it works on the recent wave of 7-inch tablets, it doesn’t currently support larger sizes.”

Encompassing smartphones and tablets means the app has to be testing on too many different screen sizes to count. In fact, here’s an image Engadget published of just some of the devices BBC used to test the Android app.

I count 13 devices in that image. Beyond handhelds, developers and testers also have to contend with OS versions. According to Android Developers’ latest count, version 2.2 and up account for nearly 98% of the Android user base. That’s pretty good coverage so it’s easy to see why BBC chose to support versions 2.2 and newer. How many versions is that you ask? Five, and all their accompanying sub-versions. Android 2.2 is nicknamed Froyo, and sticking with the alphabetical desserts theme, Android is now on Jelly Bean. That means this app works on (and was presumably tested on) Froyo, Gingerbread, Honeycomb, Ice Cream Sandwich and Jelly Bean. If you’re being thorough, each version needs to be tested on multiple devices to see how each combination reacts.

So for this one app the BBC had to test on a slew of different devices, across the major carriers in the UK and on five different OS versions. That’s what makes mobile application testing so complicated. (And why in-the-wild testing makes it some much easier.)