A guide for planning and running a moderated remote user test

There are plenty of ways you can run a usability test. Most people would advise you to do them in person; others may advise you to use a service that will automate the test. This may not always be an option for your project, so running a moderated remote usability test is the next best thing!

Remote testing can lead to incredibly helpful insights. You get to experience user’s reactions on the fly; and access your own customer base—not a panel of professional user testers. Just like in-person testing, you need to prepare ahead of time, and that prep work is a little more difficult since technology is involved.

This article is meant to help you prepare for a remote user test by highlighting the key actions to consider:

  • Recruiting your own participants
  • Technical check-Ins
  • Scheduling
  • Managing incentives
  • Getting to know the user

There will be moments where participants may back out at the last minute, you’ll run behind schedule, or your software won’t work they way you hoped it would. That’s okay! Because this guide will prepare you for anything that gets thrown your way.

Recruiting Your Own Participants

Before you can run a usability test you need to find qualified participants. There are several ways to find the right candidates for your test: use a professional recruiter; recruit from your email subscribers; asking your Twitter and Facebook followers; or recruiting customers directly from your website.

For a recent user test on a client project we chose to use Ethnio, a remote recruiting tool,because they had a very specific user base and loyal customers that we needed to target. Ethnio allowed us to place a screener on the client’s website and ask questions that would help us rule them in or out.

When determining whether or not a participant is qualified for the test, ask yourself the following questions when evaluating their responses.

  • Does the participant have anything to do with the development or design of what you are testing?
  • Do they represent your personas that you developed or a target audience you are trying to reach?
  • Are they already familiar with the website and subject matter?

Once you have your participants ready you should plan to conduct a technical check-in.

Preparing for Technical Check-Ins

Before you run the usability test I recommend conducting a technical check-in prior to the big day. Asking a participant for an extra phone call prior to the session can be seen as a waste of time. It’s important to remind the user why they signed up and give some details so they know what to expect during their testing session.

In order to make the testing event successful we know we need to do a 15 minute dry run a few days before the test to check the technology. We adjusted the language of our email to let the participant know what we expected of them. Here is an example email template we used for a recent test:

Hello [Name],

Thank you for signing up to participate in our research for X website! We can’t wait to hear your thoughts about what we’re working on.

We will reach out separately to schedule your 60 minute research session on Tuesday 12/8. During this session you will explore our site and get a $100 Visa gift card for your feedback.

But first, we wanted to do a quick technical check-in with you in order to test the software we’ll be using for the session. Are you available for a 15 minute technical check-in on Friday, 12/4 between the hours of 9:00am EST – 4:00pm EST?

There are a few things you will need to prepare for this tech check in.

1. We'd like to make sure that you have access to a web browser like Google Chrome. If you don't have access to it already, please download it before our call here: https://www.google.com/chrome/.

2. Once you have Google Chrome installed, please download the following extension so [moderator's name] can view your screen: We will be using GoToMeeting to share screens. You can go to this link and click “Add to Chrome” to install the extension.

Thank you for your time and feel free to ask any questions.

Best,

[Moderator’s Name]

Conducting the Technical Check-Ins

For this testing session we used GoToMeeting to share and record screens. Many of the users weren’t familiar with GoToMeeting. We needed to make sure they had the proper plug-in installed and knew how to share their screen. Taking 15–20 minutes to do this saves time and decreases the chances of things going wrong during the actual testing session.

For those who didn’t end up doing a technical check in, we spent a lot of time trying to make sure the audio and screen sharing was working properly. There were times GoToMeeting just kept freezing or the user couldn’t hear me. During another session, a user didn’t realize that they needed to be on their computer in order to walk through the prototype. Running a technical check-in helps avoid these issues.

If you plan on using GoToMeeting (GTM) for user testing, you can follow these steps during your technical check in.

  1. A few minutes before the scheduled technical check-in time, open the GTM desktop app and log in to the meeting. You have to manually enter the meeting number.
  2. Dial into the audio via the phone then put it on speakerphone and mute the phone to dial-in your audio code. Remember to un-mute when your user gets on the line.
  3. You can see when your user enters the meeting as their name will appear in the participant list in the GTM app. You can also see whether they are connected to the phone or computer audio by the icon next to their name in the app.
  4. After thanking the participant, walk them through the screen share setup if they haven't connected already.
  5. Press “Change Presenter” in the GTM desktop app. Normally, this will prompt them to download the desktop app, but if they have the Chrome extension installed, they won’t have to. Remind them to share their entire screen, not just the GTM screen.
  6. If you want to also see their face and they’ve agreed to using a webcam
  7. Make sure their web cam feed and their screen are visible on your desktop.
  8. Send them a URL via the chat function (something neutral like google.com) to make sure they can find and use chat. This way you’ll be able to send them links.

Now that you’ve done your technical check-ins, you can focus on scheduling the actual user testing sessions.

Scheduling

You should dedicate entire work days to user testing. That way you get in the flow of testing and won’t get interrupted by other tasks. For one of our projects, I dedicated two days to user testing with four sessions each day. Each session was an hour and I added time in between each session. When scheduling tests it’s helpful to give yourself 15–30 minutes in between sessions for you to debrief with your team. Take this moment to review your notes and figure out if there are any questions you would like to ask differently or explore different task during the next round of testing.

Since you are using an hour of someone's time to conduct research, it’s important to provide an incentive to show your appreciation for agreeing to participate in your study.

Managing Incentives

Ethnio allowed us to use their site to deliver Visa gift card codes to the participants. When it comes to determining the value of the incentive, consider the value of the insights they will give you and how hard it was to recruit participants. If you know that the project you are working on has a very specific and unique user set, you may want to give them a larger incentive. Typically, we like to provide an $50 incentive for 30 minutes; $100 for 60 minutes. Don’t forget to include that incentive in the screener to attract participants.

Get to Know the Participants

The participant is going to be nervous. They have to speak to a complete stranger, answer non-stop questions and be recorded for an hour. In order for them to be comfortable and truly say what’s on their mind, the moderator needs to move the conversation past the typical “good morning” and “how’s your day going?”

Have a pre-interview script ready and ask them about their experiences with the site. Get to know why they were on the site in the first place. Some questions we like to ask are:

  • Name
  • Age
  • Occupation, if relevant
  • How they discovered the site?
  • What were they trying to do on the site? Read an article or purchase an item?
  • What is their level of knowledge of what the site provides?
  • How often do they go to the site?

Then ask them questions based on their answers. If they mention they were purchasing a specific item, ask them why or for whom. Following up based on their answers can help you figure out what experience the user was having on the site. From there on it becomes easier to walk through the prototype, make a few jokes and really hear about their experience with the design.

Once you follow these steps, and get through the first hurdle of getting to know a participant, you should be ready for testing. Following all of these steps will make your remote user test go smoothly. Now all you have to focus on is developing your script to facilitate the conversation. Good luck and get to testing!