Three questions after a usability test

Right after a usability session, I have about ten minutes before the next one starts, and those minutes are the most honest part of my analysis. By evening I have a tidy story. By the next morning the story has swallowed the awkward bits. So I write down answers while the session is still ringing in my ears.

I keep a three-part list for this. It sits next to the longer list I use in interviews, and I number it from 3 because questions one and two are always “what happened” and “what did you expect”.

3. What did they do that I did not expect?
4. Where did they hesitate for more than a few seconds?
5. What did I say that I should not have said?

The last one stings. In a session on Thursday I said “Right, so that’s the button” when a participant had gone quiet. She had been about to find it. I took away her chance to be confused in a useful way, and I will never know what she would have said.

What I do with the answers

Each answer goes on a sticky in a shared board, one sticky per question per participant. By the fifth session the same stickies start to repeat. In the most recent round of eight sessions, “hesitated” showed up on the same screen in six of them, and I stopped writing it down after the fifth.

I try not to write solutions on the stickies. Anselm teases me that my stickies sound like a nature documentary: “Participant approaches the dropdown. Pauses. Retreats.” That is on purpose. The observation should be dull enough that nobody argues about it.

Time

The ten minutes are sometimes only five, when a session runs over. Five is enough for question three. I would rather have one honest answer than three polished ones.