The art of subjectivity and the science of objectivity in mobile usability testing


Some trace the history of usability testing to the work that Xerox did early in the 1980s while others trace usability testing to a history outside the software world, starting off with testing for military equipment. Over the years usability testing has gained a lot of prominence and is now an accepted niche in the field of software testing. However, as the discipline evolves, so do the challenges in varied testing types. One such recent challenge that this specific type of testing, has been facing is how to carry on mobile usability testing. While practices from a regular web application usability testing can be carried forward to a mobile application too, there are some specific challenges in the mobile space that call for an art to support the subjectivity and for science to support the objectivity in conducting a mobile usability study.

Firstly, it is absolutely clear that mobile usability testing is inevitable – with an increasing global user base for most mobile applications and the number of mobile device options on which applications are supported, it is best to get the product’s usability feedback from the horse’s mouth directly – in this case the actual user base. Typically, usability studies have been conducted in usability labs which are equipped with the required infrastructure, such as video recorders, audio microphones, a split room to accommodate the tester (aka user) and the usability expert who is talking to the tester to elicit specific usability feedback. This kind of formal usability testing is taking more of a back seat in the current day, where more mobile application usability testing is required. This includes studies where the user performs the test at his convenience and fills up a usability questionnaire to provide feedback, has a usability feedback session with the usability expert sharing his experiences, or sometimes even provides his feedback in a more subjective manner in a free flow style of writing. This is more so the case because the user is often not co-located with the usability expert, unlike in the web application usability testing done in the past. Also, given the sensitivity in gathering feedback from users, even if the usability expert is co-located, they are beginning to rely more heavily on a conversation style of gathering feedback as opposed to making it very formal. To this extent mobile application usability testing remains subjective and an art holding the user’s perception very high in understanding the product’s ease of use or lack thereof.

On the other hand, a tester is always striving to bring in objectivity into the overall test effort and usability is no exception. A few challenges in the mobile world are around screen resolution and associated recording options. Given the small screen size, it may first be worth to give the user a paper prototype or wireframe of the application to provide overall design and intuitiveness feedback. This does not by-pass verifying the implementation efficiency on the mobile screen, but is done with the hope to enable the user understand the flow and then further provide feedback if the flow has been implemented well within the constraints of the mobile screen. Also, recorder options are more prevalent in standard web applications that in mobile applications. Mobile sleds are increasingly being used (whether they are professional sleds or DIY sleds) to support a small camera in your recording process of both the screen and the user expressions. Outputs from such recordings are then analysed by usability experts to draw conclusions that they would have, from a typical in-lab usability test effort. Live usability tests are also being done using video solutions such as Skype to encourage live discussions between the user and the usability expert who is able to steer the conversation to get valuable yet neutral usability feedback from the user. These elements show that the science in eliciting objective usability feedback is certainly not lost with the surge in mobile applications. Indeed, what is becoming important is to draw the right balance in leveraging both the scientific and artistic techniques in getting valuable user feedback that continues to be difficult to get from an in house team of people who play the role of “software testers” regardless of any amount of skill or expertise they may possess in the field of usability.