Researching Usability

Posts Tagged ‘recruitment

Now that the usability testing has been concluded, it seemed an appropriate time to evaluate our recruitment process and reflect on what we learned. Hopefully this will provide useful pointers to anyone looking to recruit for their own usability study.

Recruiting personas

As stated in the AquabrowserUX project proposal (Objective 3), the personas that were developed would help in recruiting representative users for the usability tests. Having learned some lessons from the persona interview recruitment, I made a few changes to the screener and added a some new questions. The screener questions can be seen below. The main changes included additional digital services consulted when seeking information such as Google|Google Books|Google Scholar|Wikipedia|National Library of Scotland and an open question asking students to describe how they search for information such as books or journals online. The additional options reflected the wider range of services students consult as part of their study. The persona interviews demonstrated that these are not limited to university services. The open question had two purposes; firstly it was able to collect valuable details from students in their own words which helped to identify which persona or personas the participant fitted. Secondly it went some way to revealing how good the participant’s written English was and potentially how talkative they are likely to be in the session. Although this is no substitute for telephone screening, it certainly helped and we found that every participant we recruited was able to talk comfortably during the test. As recruitment was being done by myself and not outsourced to a 3rd person, this seemed the easiest solution at the time.

When recruiting personas the main things I was looking for was the user’s information seeking behaviour and habits. I wanted to know what users typically do when looking for information online and the services they habitually use to help. The questions in the screener were designed to identify these things while also differentiate respondents into one type of (but not always exclusive) persona.

Screener Questions

The user research will be taking place over a number of dates. Please specify all the dates you will be available if selected to take part

26th August |27th August | 13th September | 14th September

What do you do at the university?

Undergraduate 1st |2nd |3rd | 4th | 5th year| Masters/ Post-graduate | PhD

What is your program of study?

What of the following online services do you use when searching for information and roughly how many hours a week do you spend on each?

Classic catalogue | Aquabrowser catalogue | Searcher | E-journals | My Ed | Pub Med | Web of Knowledge/Science | National Library of Scotland | Google Books | Google Scholar | Google | Wikipedia

How many hours a week do you spend using them?

Never|1-3 hours|4-10 hours|More than 10 hours

How much time per week do you spend in any of Edinburgh University libraries?

Never|1-3 hours|4-10 hours|More than 10 hours

Tell me about the way you search for information such as books or journals online?

Things we learned

There were a number of things that we would recommend to do when recruiting participants which I’ve listed below:

  1. Finalise recruitment by telephone, not email. Not surprisingly, I found that it’s better to finalise recruitment by telephone once you have received a completed screener. It is quicker to recruit this way as you can determine a suitable slot and confirm a participant’s attendance within a few minutes rather than waiting days for a confirmation email. It also provides insight into how comfortable the respondent is when speaking to a stranger which will affect the success of your testing.
  2. Screen out anyone with a psychology background. It is something of an accepted norm amongst professional recruitment agencies but something which I forgot to include in the screener. In the end I only recruited one PhD student with a Masters in psychology, so did not prove much of a problem in this study. Often these individuals do not carry out tasks in the way they would normally do, instead examining the task and often trying to beat it. This invariably can provide inaccurate results which aren’t always useful.
  3. Beware of participants who only want to participate to get the incentive. They will often answer the screener questions in a way they think will ensure selection and not honestly. We had one respondent who stated that they used every website listed more than 10 hours a week (the maximum value provided). It immediately raised flags and consequently that person was not recruited.
  4. Be prepared for the odd wrong answer. On occasion, we found out during the session that something the participant said they had used in the past they hadn’t seen before and vice versa. This was particularly tricky because often students aren’t aware of Aquabrowser by name and are therefore unable to accurately describe their use of it.

Useful resources

For more information on recruiting better research participants check out the article by Jim Ross on UX Matters: http://www.uxmatters.com/mt/archives/2010/07/recruiting-better-research-participants.php. There is also a similarly useful article by Abhay Rautela on Cone Trees with tips on conducting your own DIY recruitment for usability testing: http://www.conetrees.com/2009/02/articles/tips-for-effective-diy-participant-recruitment-for-usability-testing/.

Have I missed anything? If there is something I’ve not covered here please feel free to leave a comment and I’ll make sure I respond. Thanks

Advertisements

In a previous blog I evaluated the progress of the data gathering stage of persona creation for both Aquabrowser UX and UX2.0. As the data gathering has now been completed and analysed, we have the beginnings of our personas. It therefore seemed a good time to reflect on the process as well as document and review our methods. In the first of three blogs detailing our persona creation, I will first talk about the data gathering methods and reflect on its success.

Originally the plan had been to create personas by conducting qualitative research and validating the segments with quantitative data. Unfortunately we underestimated the time taken and resources required to conduct the qualitative research and as such were unable to validate the personas using quantitative research. Although this approach is good when you want to explore multiple segmentation models and back up the qualitative findings with quantitative data, personas created without this extra step are still valid and extremely useful. As this is the first time the team has conducted persona data gathering, it took longer to do than anticipated. Coupled with the restrictions on time and budget for this project, the additional validation was always an ambition. I’ve stepped through the process we used below to allow others to adopt it if needed. The process is a good template for conducting all types of interviews and not just to create personas.

1. Designing the interviews

When designing the interview questions the team first set about defining the project goals. This was used as a basis for the interview questions and also helped to ensure that the questions covered all aspects of the project goals.

Goal 1: In relation to University of Edinburgh digital library services, and AquaBrowser, identify and understand;

  • User general profiles (demographic, academic background, previous experiences)
  • User behaviours (e.g. information seeking) / use patterns
  • User attitudes (inc. recommendations)
  • User goals (functional, fit for purpose)
  • Data, content, resource requirements

To keep the interview semi-structured and more conversational, the questions created were used primarily as prompts for the interviewer and to ensure that the interviewees provided all the information being sought. More general questions were posed as a springboard for more open discussion. Each question represented a section of discussion with approximately six questions in total. Each question in turn had a series of prompts. The six opening questions are detail below:

  1. Could you tell me a bit about yourself…?
  2. Thinking now about how you work and interact with services online, what kind of activities do you typically do when you sit down at the computer
  3. I want to find out more about how you use different library services, can you tell me what online library services you have used?
  4. We want to know how you go about finding information…What strategy do you have for finding information?
  5. Finally, we’d like to ask you about your own opinions of the library services..  a. What library or search services are you satisfied with and why? b. Why do you choose <mentioned services> over other services?

Interviewees were also given the opportunity at the end of the interview to add anything they felt was valuable to the research or which they just wanted to get off their chest. Several prompt question were modified or added to the librarian interview script, otherwise the overall scripts were very similar

When the interview was adapted into a script for the interviewer, introductory and wrap-up sections were added to explain the purpose of the interview and put the interviewees at ease. These sections also provided prompts to the interviewer to ensure permission was obtained beforehand and that the participant was paid at the end.

2. Piloting the interview

The script was piloted on a colleague not involved in the project a few days before the interviews began. This provided an opportunity to tweak the wording of some of the questions so they were clearer, time the interview to ensure it could be conducted in approximately 45 minutes and also help the team to become more familiar with the questions and prompts. Necessary changes were consequently made to the script to be used for the first ‘real’ interview.

3. Recruitment – Creating a screener

In order to recruit a range of users at the university, a screener was devised. This would provide information on each participant’s use patterns and some basic demographic details. It also allowed us to find out the availability of each participant as the interviews were intended to be conducted over a four-week period in June and July. It also made it easier to collect contact details from users who had already agreed to take part. As with most user research where incentives are involved, there is always the danger that participants will be motivated by the reward of payment and consequently will say whatever they need to say in order to be selected. As we were looking for users who were familiar with Aquabrowser and Voyager (The ‘Classic’ catalogue), we disguised these questions among other library services. This prevented the purpose of the research from being exposed to the participant. The screener questions we used are detailed below:

Screener questions:

  1. Please confirm if you are willing to take part in a 45 minute interview? Note: There will be a £15 book voucher provided for taking part in an interview.
  2. In addition to interviews we are also recruiting for website testing sessions (45-60 min). Would you would be interested in taking part?
    Note: There will be a £15 book voucher provided for taking part in a website testing session.
  3. What do you do at the university? Undergrad: 1st/2nd/3rd/4th/Post grad/PhD/Library staff/Teaching staff/Research staff/other.
  4. What is your department or program of study?
  5. Which of the following online services do you use at the University and how many hours a week do you spend on each? Classic catalogue/ Aquabrowser catalogue/Searcher/E-Journal search/ PubMed/ My Ed/Web of Knowledge/Science/Science Direct.
  6. How much time per week do you spend in any of Edinburgh University libraries? None/Less than 1 hour a week/1-3 hours a week/4-10 hours a week/More than 10 hours a week.
  7. Please state your prefered mode of contact to arrange interview date/time.
  8. Please leave your name.
  9. Please leave relevant contact details: Email and/or telephone number.

Thank you very much for your time.  If you are selected to participate in the current study, we’ll be in touch to see when would be the best time for your session.

4. Recruitment – Strategy

A link to the screener was publicised through a variety of streams. An announcement was created and placed in the MyEd portal which every person within the university has access to (staff and students). In addition to this, an event was created which was also visible within the events section of MyEd. Several email invitations were sent via mailing lists requesting participation. These lists included the School of Physics, Divinity and Information Services staff.

To encourage students and staff to participate an incentive was provided. A £15 book voucher was promised to those who agreed to take part in an interview. The screener was launched on 21st May and ran until the interviews were completed on 15th July. Interviews were scheduled to take place over four weeks which began on 17th June. Six interviews were carried out on average each week, taking place over two separate days. These days varied, but often took place on Tuesdays, Thursdays and Fridays. This was influenced by the availability of team members to carry out the interviews. Each participant was given the opportunity to name their preferred location for the interview. Those interviews that can take place in the user’s own environment are more likely to put the participant at ease and consequently produce better interviews. However, every participant ended up opting to meet at a mutually convenient location – the main library. This public venue is familiar to all participants and centrally located making it less intimidating and easy to find. It also enabled more interviews to be conducted over a short period of the day as travelling to various locations was not required.

Participants were recruited based on a number of factors. Their position in the university (student, staff etc.), their familiarity (or in some cases not) with library services, especially Aquabrowser and Voyager (Classic catalogue). Individuals who spent a reasonable amount of time using these services were of interest but a number of individuals who did not spend much time using the services were also recruited to provide comparisons. Obviously their availability was also an important factor and anyone who was not available in June and/or July were excluded.

Although the screener speeded up the recruitment process, there was still a number of individuals on the list who did not respond to additional email requests to participate. This is always frustrating when people have apparently registered their interest when completing the screener. Despite this we managed to recruit 19 participants from a list of 82 respondents which was approximately a 23% response rate. Unfortunately from these 19 individuals, two individuals dropped out at the last-minute. One person did not show up and another cancelled due to ill-health. As these cancellations occurred on the last day of interviews and did not represent a under-represented demographic group, the decision was taken not to recruit replacements and to conclude the data gathering stage with 17 participants.

Unfortunately there were some groups who were under-represented. The biggest concern was the limited number of staff and in particular,  lecturers in the study. This ultimately meant that this group could not be adequately analysed. Time limitations meant it was difficult to undertake additional strategies to target these individuals. The data gathered was only able to provide personas representing those interviewed and consequently a persona for faculty staff was not possible. Any future persona development work should ensure that a variety of lecturers and researchers are interviewed as part of the process.

5. Conducting the interviews

Before the interviews began, several preparations had to be made. Booking audio recording equipment, sourcing a digital camera for images and creating consent forms for both audio and visual documentation was done. Two team members were present for every interview. One would take notes while the other interviewed the participant. These roles were swapped for each new interview, giving both team members the chance to be both interviewer and note-taker. After discussing how best to take notes it was decided that having a printed template for each interview which the note-taker could complete would be a good strategy. This would help to keep notes in context as much as possible and make the note-taking process as efficient as possible. Doing so removes the danger of important information being lost. The note-taker would also record time stamps each time something ambiguous or important was said so that clarification could be made later by listening to the recordings.

After each interview the team transferred the notes onto the project Wiki. Doing so allowed a post-interview discussion to take place where both members reflected on their understanding of the interviewees comments. It also provided the opportunity to review the interview and discuss improvements or changes that could be made to the next one. This was particularly useful for the first few interviews.

Having two team members present for each interview was time-consuming but also provided many benefits. During the interview it gave the interviewer a chance to consult with someone else before concluding the interview. Often the note-taker may have noted something the interviewer overlooked and this time at the end ensured that any comments made earlier could be addressed. In addition, any missed questions or prompts were asked at this point. It is also beneficial for all team members to be present when it comes to analysing the data at the end. Individuals do not have to rely heavily on the notes of another to become familiar with the participant. This is particularly important when it is not possible to have transcripts of each interview made. Finally, having a note-taker removes the burden from the interviewer who does not have to keep pausing the interview to note down anything that has been said. As these interviews were intended to be semi-structured and have a discursive feel, having a note-taker was crucial to ensuring that this was achieved.

Conclusion

Overall the interviews were quite successful. However, in future more interviews with staff should be conducted in addition to a web survey so that the segmentation can be validated. Full access to the documents created during the process and the resources consulted throughout can be found on the project Wiki.

In the second part of the series guest blogger, Liza Zambolgou will be discussing the segmentation process and how we analysed all of the data collected from the interviews.


del.icio.us bookmarks

Twitter feed

Archive