Usability Test Observation
Time for a usability-related blog…
A usability test is an excellent opportunity for the development team to see someone use (or *try* to use) their product. For a usability test facilitator, having observers in the testing room can increase the workload as you have to manage both the test and the observers. A good briefing can keep observers in line, but the facilitator still needs to be on his or her toes.
The obvious solution is to get the observers out of the room yet still let them see. Traditional usability labs have observation rooms behind 1-way mirrors, but most companies can’t afford such luxury. Enter technology. You can transform any meeting room into an observation room with a computer and projector.
This setup works really well for remote usability testing. We just started using WebEx to conduct usability tests (Creo already uses WebEx for other uses, so the usability work gets it for “free”). Remote testing has its limitations (IBM has a good article on the topic — part 1 and part 2) but it makes remote observation that much easier. For local testing you can do the same thing with products such as Morae.
You need a computer, projector, and speaker phone for the observation room. Set up the meeting room 30 minutes prior to the test session. Power on the computer and projector and connect to the remote meeting session. For WebEx, we use the phone (instead of VoIP) so I will also connect the observation room’s speaker phone to the teleconference line I’ve reserved in advance. Put the phone on mute and stick a note on it warning people not to un-mute it. WebEx requires a user name when connecting to a meeting so I use “Observation Room”. I tell the participant at the beginning of the session that some members of the development team are observing the session. I am not specific about how many observers there are and I’ve never been asked.
Other technologies may differ, but when using WebEx and standard teleconference systems, it’s important to force all observers to use the observation room and not connect to the meeting individually. I’ve found that observers often can’t spend an entire two, sometimes three, hours watching one session. An individual leaving a telecon and WebEx session is very obvious and disruptive (all sorts of beeps go off), but someone walking out of the observation room is silent. Also, the abstract “Observation Room” participant seems less intimidating than 3, 5, or 8 online individuals.
Another tip when facilitating these tests is: don’t use your phone’s conferencing feature. It makes it impossible to manage dropped lines and is a burden you can easily avoid.
Finally, you’ll still want to brief your observers on proper etiquette. Among other things, I remind them that the phone is to remain on mute and they should not problem solve during the test. To keep them focused on the test, I include a copy of the test’s scenarios, structured note-taking sheets, and copies of any paper material the tester has access to. If possible, you may want to include someone who can facilitate the group.
March 9th, 2005 at 7:10 am
This is interesting. It implies that all developers can and should take an interest in the usability of the product they are developing. I suppose for the area of usability you are talking about, it is the applications level programmers who are writing code that directly interfaces with the user that you’re targeting to be observers.
[In my case](http://www.flipstart.com), the majority of the usability issues we have are related to physical usability of the device. How big are the keys? Where is the touchpad? Are the audio jacks accessible for the user models we have defined? Are the user models we have defined even accurate? In the end, I often feel that the net results from *our* usability studies are counter-intuitive (to me) and perhaps seem somewhat arbitrary. I suppose this could be partly because I am not in the target market (ie., I am abnormal), or it could just be bad user studies.
In the end, because it seems arbitrary, it makes me care less (sorry Jay) about the usability, and more like “just tell me where to put all this stuff in the box so I can design your product”. Do you think observing the user studies will cure me of my bad attitude?
March 13th, 2005 at 9:14 pm
One of the other things usability engineers/specialists/whatever should do is get buy in from the design team that they are testing the correct things. We chatted about this issue and it sounded like there were many unknowns regarding how the FlipStart will be used. If the test focuses on the incorrect use case, the results are useless.
The best thing I can think of doing if I had to tackle a novel product like that is to do longer-term studies and see what kind of usage patterns result. For example, do a kind of diary study where you outfit a number of participants with prototype devices and track how they use them day to day. (There are many ways to do this from following them around to getting them to record their usage in a journal or voice recorder…)