Researching Usability

Posts Tagged ‘productivity

When carrying out usability studies on search interfaces, it’s often better to favour interview-based tasks over pre-defined ‘scavenger-hunt’ tasks. In this post I’ll explain why this is the case and why you may have to sacrifice capturing metrics in order to achieve this realism.

In 2006, Jared Spool of User Interface Engineering wrote an article entitled Interview-Based Tasks: Learning from Leonardo DiCaprio in it he explains that it often isn’t enough to create test tasks that ask participants to find a specific item on a website. He calls such a task a Scavenger-Hunt task. Instead he introduces the idea of interview-based tasks.

When testing the search interface for a library catalogue, a Scavenger Hunt task might read:

You are studying Russian Literature and your will be reading Leo Tolstoy soon. Find the English version of Tolstoy’s ‘War and Peace’ in the library catalogue.

I’ll refer to this as the Tolstoy Task in this post. Most of your participants (if they’re university students) should have no trouble understanding the task. But it probably won’t feel real to any of them. Most of them will simply type ‘war and peace’ into the search and see what happens.

Red routes

The Tolstoy Task is not useless, you’ll probably still witness things of interest. So it’s better than having no testing at all.

But it answers only one question – When users know the title of the book, author and how to spell them both correctly, how easy is it to find the English version of Leo Tolstoy’s War and Peace?

A very specific question like this can still be useful for many websites. For example a car insurance company could ask – When the user has all of his vehicle documents in front of him, how easy is it for them to get a quote from our website?

Answering this question would give them a pretty good idea of how well their website was working. This is because it’s probably the most important journey on the site. Most websites have what Dr David Travis calls Red Routes – the key journeys on a website. When you measure the usability of a website’s red routes you effectively measure the usability of the site.

However many search interfaces such as that for a university library catalogue, don’t have one or two specific tasks that are more important than any others. It’s possible to categorise tasks but difficult to introduce them into a usability test without sacrificing a lot of realism.

Interview-based tasks

The interview-based task is Spool’s answer to the shortfalls of the Scavenger Hunt task. This is where you create a task with the input of the participant and agree what successful completion of the task will mean before they begin.

When using search interfaces, people often develop search tactics based upon the results they are being shown. As a result they can change tactics several times. They can change their view of the problem based upon the feedback they are getting.

Whilst testing the Aquabrowser catalogue for the University of Edinburgh, participants helped me to create tasks that I’d never have been able to do so on my own. Had we not done this, I wouldn’t have been able to observe their true behaviour.

One participant used the search interface to decide her approach to an essay question. Together we created a task scenario where she was given an essay to write on National identity in the work of Robert Louis Stevenson.

She had decided that the architecture in Jekyll and Hyde whilst set in London, had reminded her more of Edinburgh. She searched for sources that referred to Edinburgh’s architecture in Scottish literature, opinion on architecture in Stevenson’s work and opinion on architecture in national identity.

The level of engagement she had in the task allowed me to observe behaviour that a pre-written task would never have been able to do.

It also made no assumptions about how she uses the interface. In the Tolstoy task, I’d be assuming that people arrive at the interface with a set amount of knowledge. In an interview-based task I can establish how much knowledge they would have about a specific task before they use the interface. I simply ask them.

Realism versus measurement

The downside to using such personalised tasks is that it’s very difficult to report useful measurements. When you pre-define tasks you know that each participant will perform the same task. So you can measure the performance of that task. By doing this you can ask “How usable is this interface?” and provide an answer.

With interview-based tasks this is often impossible because the tasks vary in subject and complexity. It’s often  then inappropriate to use them to provide an overall measure of usability.

Exposing issues

I believe that usability testing is more reliable as a method for exposing issues than it is at providing a measure of usability. This is why I favour using interview-based tasks in most cases.

It’s difficult to say how true to life the experience you’re watching is. If they were sitting at home attempting a task then there’d be nobody watching them and taking notes. Nobody would be asking them to think aloud and showing interest in what they were doing. So if they fail a task in a lab, can you be sure they’d fail it at home?

But for observing issues I feel it’s more reliable. If participants misunderstand something about the interface in a test, you can be fairly sure that someone at home will be making that same misunderstanding.

And it can never hurt to make something more obvious.

In Lorraine’s last blog she described the data gathering methods used to obtain representative data from users of Edinburgh University’s Library services, the purpose of which was to identify patterns in user behaviours, expectations and motivations to form the basis of our personas. Raw data can be difficult to process and it is impossible to jump from raw notes to finished persona in one step, hence our six step guide.

There is no one right way to create personas and it depends on a lot of things, including how much effort and budget you can afford to invest. There are lots of articles on the web detailing various approaches and after much reading we decided to rely on 2 main sources of information which we felt best suited our needs.

One resource was the Fluid Project Wiki which is an open, collaborative project to improve the user experience of community source software and provides lots of useful guidance as well as sample personas. The other resource which we heavily relied on throughout the whole process was Steve Mulder’s book The User Is Always Right: A Practical Guide to Creating and Using Personas for the Web, which contains lots of great advice as well as step-by-step coverage on user segmentation.

There are three primary approaches to persona creation, based on the type of research and analysis performed:

  1. Qualitative personas
  2. Qualitative personas with quantitative validation
  3. Quantitative personas

There are a number of important steps to go through in order to get from raw data to personas and I will now explain the tools and methods used to generate our segments and personas for anyone who wishes to follow in our footsteps.

The first thing we did was to plan out a schedule of work which consisted of the following:

  1. Review and refine interview notes in the project wiki and flesh out user goals
  2. Write summaries for each of the participants
  3. Do a Two by Two comparison, to identify key similarities/differences
  4. Identify segments
  5. Write the personas
  6. Review personas

Step 1: Review/refine notes

We spent a day reviewing our notes in the wiki and fleshing out goals by referring to written notes taking during each interview, checking the audio recordings where necessary. We worked as a team which was beneficial as we were both present for each interview and therefore had a good grasp of all the data in front of us. Once we were happy with our set of notes, we printed out participant’s interview notes and attached each to the white board to make it easier to review all data grouped together.

Step 2: Summarise participants

Next step was to summarise each of our 17 participants (try to figure out who are these people) based on the following 4 categories.

  • Practical and personal goals
  • Information seeking behaviour
  • How they relate to library services
  • Skills, abilities and interests

We used different coloured post-it notes to denote each of the above categories. Once we had gone through this process for each participant, our whiteboard was transformed into a colourful mirage of notes.

We were now ready to start a two-by-two comparison of participants.

Step 3: Two-by-Two Comparisons

The next step utilised the two-by-two comparison method, a technique advocated by Jared Spool at User Interface Engineering ( This works by reading 2 randomly chosen participant summaries and listing attributes that make the participants similar and different. We then replaced one of the summaries with another randomly chosen one and repeated the process until all summaries were read.

Below is a list of some of the distinctions identified between our participants, using this method:

  • Type of library user
  • Years at Edinburgh University
  • Use of Edinburgh University library resources (digital and physical)
  • Use of external resources
  • System Preference (Classic or Aquabrowser)
  • Attitude to individual systems
  • Information seeking behaviour

We then created a scale for each distinction identified during the two-by-two comparison and determined end points. Doing so allowed us to place each participant on the scale and directly compare them.  Most variables can be represented as ranges with two ends. It doesn’t matter whether a participant is a 7 or 7.5 on the scale; but what matters is where they appear relative to other participants. The image below provides an example of our 12 scales mapped for each of our 17 participants.

Step 4: Identify Segments

Now that we had all our participants on the scales, we then colour coded each individual to make it easier to identify groupings of participants on each of the scales.  We looked for participants who were grouped closely together across multiple variables. Once we found a set of participants clustering across six or eight variables, we saw this as a major behaviour pattern which formed the basis of a persona.

After quite a bit of analysis, we identified 6 major groupings, each identifying an archetype / persona, which we gave a brief description to on paper, outlining the characteristics and identifying their unique attributes.

After reviewing each description we realised that group 6 was very similar to group 4 and so merged these two sets together, leaving 5 groups at the end of this step.

When carrying out this step, it is important to remember that your groups should:

  • explain key differences you’ve observed among participants
  • be different enough from each other
  • feel like real people
  • be described quickly
  • cover all users

Step 5: Write the Personas

We were now ready to write up our 5 personas.  For each group we added details around the behavioural traits based on the data we had gathered, describing their goals, information seeking behaviours and system usage amongst other things. We also talked about frustrations and pain points as well as listing some personal traits to make them feel more human.

We gave each persona a name and a photo which we felt best suited their narrative.  We tried to add parts of participant’s personalities without going overboard as this would make the persona less credible.  We kept the detail to one page and based it on a template provided by the Fluid Project wiki. It’s important to keep persona details to one page so they can be referred to quickly during any discussions. Remember that every aspect of the description must be tied back to real data, or else it’s shouldn’t be included in the persona.

Some people prefer to keep their persona details in bullet points, but we felt that a narrative would be far more powerful in conveying each of our persona’s attitudes, needs and problems.  We also added a scale to each persona, detailing their behaviour and attitudes, which serves as a visual summary of the narrative and main points.  It may be useful to refer to Fluid Persons Format page for example of these templates:

Step 6: Review the Personas

Once our personas were written, we reviewed them to ensure they had remained realistic and true to our research data. We felt that 2 personas in particular had more similar behaviours and goals than differences so we merged them into one complete persona. This left us with 4 library personas representing the students and librarians who were interviewed:

  • Eve the e-book reader: “I like to find excerpts of books online which sometimes can be enough. It saves me from having to buy or borrow the book.”
  • Sandra the search specialist: “In a quick-fire environment like ours we need answers quickly”
  • Pete the progressive browser: “Aquabrowser and Classic, it’s like night and day”
  • Baadal the search butterfly: “Classic is simple and direct but Aquabrowser’s innovative way of browsing is also good for getting inspiration.”

A full description of the personas can be found on the persona profiles page of our project wiki:

Research has shown that a large set of personas can be problematic as the personas all tend to blur together. Ideally, you should have only the minimum number of personas required to illustrate key goals and behaviour patterns, which is what we ended up with. Finally, to ensure we had a polished product, we asked a colleague who was not involved in the persona creation, to review the personas for accuracy in spelling and grammar.


From my experience, I would say that the most difficult step of the process was getting from step 3 (Two by Two comparison) to step 4 (Identify segments). Although we had initially planned to spend 3 days creating our personas, in the end it took us 5+ days.  If we were to repeat this exercise, I would allocate adequate time directly after each individual interview to write up detailed notes on the interviewee, detailing their specific goals, behaviours, attitudes and information seeking behaviour, rather than waiting until a later date to review all the notes together, as described in Step 2. In saying this, there are various different approaches which can be taken when creating personas and we would be very interested to learn what other researchers might do with the same data.

In the concluding part of this blog series, “User Research and Persona Creation Part 3: Introducing the personas”, Lorraine will discuss how we plan to keep the personas relevant and current in the future. bookmarks

Twitter feed