Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

How to Conduct a Usability test on a Mobile Device

Jeff Sauro • May 1, 2012

When it comes to testing websites there are many unmoderated and moderated solutions.

But if you've ever tried to evaluate an app or website on a mobile phone or tablet there are fewer options.

Usertesting.com offers a new mobile testing service which recruits users and records their mobile devices while they interact with your app or website.

Tools like Userzoom and Loop11 also collect some data from users on their devices.

But for an in-person moderated usability test on a mobile device, here's the setup that we've found works well.

What you'll need:
  1. The MOD 1000 or other macro-focused camera
  2. Facilitator Computer
  3. Screen Recorder : GoTo Meeting, Camtasia or CamStudio
  4. Computer to administer tasks  to the user
  5. External Microphone
  6. Webcam (Optional)


We have our usability lab positioned with the facilitator and user facing each other in a perpendicular arrangement (see the diagram below). This allows for easy face-to-face communication and the occasional intervention when the user has problems with their computer or device.



Recording the Screen

Let me save you some research. As of spring 2012 there isn't a way to record the real-time output of an iPhone or Andoid device without rooting or jail breaking the device (something most users won't want you to do to their phone).  There isn't a Camtasia or GoTo Meeting solution for mobile devices that allows you to share and record the screen.

However, even when the technology becomes readily available like on desktops, it probably wouldn't be sufficient. When we record the screen of someone's computer we can see the mouse movements and keystrokes so a simple output of the monitor captures almost all the relevant interactions.

On a mobile phone it's the fingers that do the talking instead of the mouse pointer. What's more, mobile phones are well, mobile, so users will move and rotate phones all the time. You'll want to capture as much of that as possible. This means you'll need an external camera. 

We've tested many users and tried many setups with different combinations of cameras and rigs. The one that works the best is a camera and rig we developed ourselves—the MOD 1000.


The MOD 1000 (Mobile Observation Device) is our lightweight rig for recording mobile screens and interactions.  The macro focused camera is mounted on a lightweight aluminum plate with a grip-tight surface. The image of the user's phone stays still while the user moves and manipulates their device instead of being confined to laying it flat on a desk.



The software works well on Macs and PCs. The MOD 1000 allows for landscape or portrait displays and works on all size smart phones.  The cradle is light and our users seem to stop noticing they're holding it after a few minutes. The camera on the MOD 1000 plugs right into the facilitator's computer, so the facilitator can see exactly what the user is doing in high resolution.

Displaying Task Scenarios

We use UserZoom to display task-scenarios in randomized order and collect post-task/post-test questions along with task-times.  We create a background screen that has the most important global information for the user throughout the test--such as WiFi username and password and the test URL. Often we are testing both a client's mobile and desktop interface so having the tasks in a tool like UserZoom allows us to more easily collect data on the same tasks for both mobile and desktop interfaces.

Sharing the Screen

We have found that it is necessary to share the screen from the participant's computer with the facilitator's computer.  We use Join Me, a free lightweight web application that allows for screen and mouse sharing. Just download the app on the user's computer and start a screen share. On the facilitator's computer, go to join.me in your web browser, type in the code from the user's computer, and voila, you can see exactly what the user sees. You are then able to take control of the user's mouse as needed. 

We like having control over the user's computer as we find many users forget to start the tasks as they are usually focused on using their mobile phone. When the facilitator has control of the participant's screen, the facilitator can go ahead and start the tasks if the user forgets and advance the screen as necessary. Seeing the user's screen is also helpful for knowing what task users are on when reviewing video recordings.  

Webcam

We find  it helpful to have a webcam pointed at the users face and phone with the feed going right to the facilitator's computer.  This second camera provides an alternate view which can be helpful for capturing how the user is physically interacting with the device (more so than say a stationary computer) as well as getting those informative facial expressions of frustration or delight.

Sound

We record our sessions and often have clients viewing remotely or from the observation room so having good sound is important. Because the user and the facilitator are not sitting around one computer, the built in computer microphone will probably not work all that well. We have found audio from a good webcam provides clear sound from the user and facilitator or alternatively from a good external microphone (there are many bad ones out there so try a few).

Recording the Session

We use GoTo Meeting to both share and record the sessions.  We often have clients viewing remotely already, so using GoTo meeting's built-in recording works fine.  GoTo Meeting can save the audio and video of the sessions as a compressed and sharable .WMV file that can be used for creating highlight reels and reviewing the interactions later. 

After each session, it takes GoTo meeting about 30 minutes to compress an hour of video. GoTo meeting doesn't save the web-cam of the user, so to capture the user's screen, mobile phone and desktop we use Camtasia.  



As mobile applications and websites continue to proliferate the need to quickly test and evaluate will also grow. For now, moderated "lab" usability testing is still one of the best ways to capture the rich experience of interacting with a mobile device.

We hope this quick overview of our setup will help others gather the data to continue improving the mobile user experience...we can then continue debating the best way to display mobile websites.



About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Usability Testing, Mobile Usability Testing
.

Posted Comments

There are 12 Comments

July 10, 2014 | Chiao wrote:

I plan to conduct a study of a tablet app with remote participants - moderated session. I will have the prototype running on iPad, how do I share this remotely with remote participants? 


February 28, 2014 | Anna wrote:

We use a similar setup as the one described in this article. However, we find that the refresh rate for GoToMeeting is not fast enough to let viewers watching a live session catch all the quick finger movements and gestures. Does anyone have a better screensharing solution? 


October 3, 2013 | Magitest wrote:

Magitest records usability sessions on iPhone and iPad. Screen, microphone, camera and gestures are all captured on the device so you can test in the lab or in the wild. www.magitest.com 


February 10, 2013 | Kelly Ann wrote:

A great article! rnrn@John Romadka we have been using Mr Tappy in our mobile lab for 6 months and he is a great tool, as long as you are using a light-weight camera that will not pull off the velcro he works well. However, his base is not at an angle and therefore it can be tricky for the participant to interact with their device in a natural way while trying to keep the screen in the focus range of the camera. rnrnWhen using an external camera it's ideal to be using a camera with a wide angle lens so that participants have a bit more room to move around without being restricted to holding their mobile device right under the arm and camera. We typically map out an area on the table with a felt pad or masking tape wherein the participant can move freely and still be captured by the camera. rnrnScreen capture seems to be the way to go as environmental factors such as overhead light or sunlight when testing outdoors can really spoil recordings during mobile user testing as well as frustrating participants with the glare.  


October 11, 2012 | Bryant Jaquez wrote:

This is great, but there are ways to do a lower cost test. I like to create a temporary m.example.com version of a website and install UX testing tools on that site. Depending on the project, you can use kissinsights, crazyegg, mouseflow or even split testing tools. The goal is to gather as much data as possible into how your customers interact with your mobile site.

After you have enough data, you can remove the m.example.com version of your site and build out a dynamic website on one URL. I actually wrote a whole blog post about this process http://www.brewseo.com/everything-you-need-to-know-about-mobile-seo-urls-usability-and-user-testing/ 


September 24, 2012 | Daniel wrote:

We've also tried with usertesting.com and got great results so far. We also found some other tools from http://biz30.timedoctor.com/split-testing-and-usability-tools/ and hope that we can get our testers soon so we can achieve greater results from our usability testing/ 


May 24, 2012 | Jeff Sauro wrote:

Yes, the MOD 1000 does support tablets and phones.

We have a tablet mount which the MOD fits into very easily and provides the ideal angle for portrait and landscape tablet viewing.

You can see pictures on the product page


May 15, 2012 | Mary wrote:

Does the MOD 1000 rig support tablets as well as phones? 


May 10, 2012 | Arnoud Boekhoorn wrote:

It is possible to record the screen of mobile devices without jailbreaking or rooting the device. We have been using a method described by http://www.cxpartners.co.uk/ last year (page not available anymore) and adapted it resulting in the following setup:

* iOS / Android mobile device
* HDMI cable for iOS devices
* HDMI cable for Android devices
* DVI - HDMI converter plug
* Epiphan DVI2USB Solo frame grabber
* 2x HD webcam (1x face, 1x mobile device/hands & fingers)
* Quicktime Player to show 1 webcam on the screen
* Silverback or ScreenFlow to record the screen (showing 1 webcam and the device) and the 2nd webcam

All connected to a single Mac mini. We have had great results with this setup. 


May 2, 2012 | John Romadka wrote:

There's another webcam rig that appears promising too: http://www.mrtappy.com 


May 2, 2012 | John Romadka wrote:

There's actually a new app for the Mac that just came out a few weeks ago called Reflection (http://reflectionapp.com/) that utilizes iOS video mirroring and allows you to get a live feed on your Macs desktop from any iOS device without jail breaking or installing anything on the iOS device. I testing it and works very well!

I then then fired up Camtasia and a webcam and was able to record the desktop with it displaying the live feed (video mirroring) and the webcam to see the actual users (me) interaction with the device, and you have a perfect combination (high res. feed from Reflection, and device interaction with webcam) all recorded into a single movie!

Also, I ran across something VERY interesting this week that will allow recording a "Remote" mobile usability test. There's a SDK for iOS (long awaited) that will record app interaction without being in the same location. The hitch is that you need to build the SDK into your app to record it and you can't remotely observe interactions on any other apps. But it's a step in the right direction. (http://delight.io/)  


May 2, 2012 | Nikolaj Jensen wrote:

I have done several tests on mobile devices, and I have always used a screen recording software. There are a few good ones out there that will record the screen, as well as the interactions on-screen, just like e.g. camtasie. The only caviat is that they require iPhones to be jailbroken and android ones to be rooted. Not a difficult task though. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 5: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4079 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.