Researching Usability

Archive for the ‘UX2’ Category

I’m very pleased to announce that the article written for Library Hi Tech based on research conducted earlier this year is now available. For more information and to read the article in full follow the link below. You will need to be logged in to read the full text.

http://www.emeraldinsight.com/journals.htm?articleid=1949248&ini=aob&.

– Library Hi Tech, Vol. 29 Iss: 3, pp412- 423

Many thanks to everyone who made it along to our event yesterday. I hope that it was informative and provided some guidance on planning your own user research. I’ve uploaded my slides from the morning presentation on Slideshare for those might want to bookmark it or anyone who didn’t make it along. I think it’s fair to say that based on some feedback we received, the whole day was a great success. If you were there and wish to send your feedback (good and bad) please feel free to add your comments here or drop me a line.

If you hadn’t already heard, the UCD day was the last activity from the UX2.0 project which officially ended last month. It’s been a great 24 months working on this project and while putting together the slides for the UCD Day, I realised just how much work we have produced. Hopefully all of the documents, reports and blog posts which we contributed during that time will be put to good use by others. I’m hoping to continue this blog in some capacity once I settle into a new role but fear it might go a little quiet in the short-term.

Thanks for visiting the blogs and project website/wiki and I hope you’ll continue to enjoy reading my posts in the future.

Lorraine

EDIT: I forgot to mention, those who came to the prototyping session (and those who didn’t) who are interested in trying Balsamiq themselves can do so through our pilot scheme: https://www.wiki.ed.ac.uk/display/UX2/JISC+Balsamiq+Pilot. Please remember to register as an EASE friend first to gain access. It’s a great opportunity to try Balsamiq and determine if it’s worth purchasing a licence. It really is a great tool and easy to use too!

As the UX2 project wraps up this month, we have planned an event which we hope will introduce user-centred design and usability in digital libraries to others through our case studies.

The event is free and funded by JISC meaning it’s primarily aimed at those in the education sector who are new to usability, user-centred design and user interface prototyping.

It takes place on Tuesday 17th May and the runs from 0930 – 1700

The practical session in the afternoon has a limited number of places so early registration is recommended. During the session you’ll get the chance to create your own wireframes first in paper then using the prototyping tool, Balsamiq. Our project recently funded a licence which makes Balsamiq available to others to try out. For more information on this visit the Wiki: https://www.wiki.ed.ac.uk/display/UX2/JISC+Balsamiq+Pilot

Programme:

0900-0930 Registration

0930-1030 User-centred design and faceted search user interface (Boon Low, Lorraine Paterson)
This session will introduce user experience and user-centred design concepts and how these are related to open-source development. The complexities of UIs will be highlighted through a discussion on usability issues and the existing UI design patterns in particular, faceted search.

1030-1100 Break

1100-1230 Usability and user research: case studies and current practice (Lorraine Paterson, Boon Low)
This session will introduce user research methods and highlight examples from case studies to demonstrate how these methods can be used. A guide to usability test techniques will be provided including a best practice guide and practical advice.

Lunch

1330-1700 Practical: Prototyping for beginners (max 12 participants) (Neil Allison, Liza Zamboglou, Lorraine Paterson)
This non-technical introduction to prototyping will be a mixture of presentations, discussion and practical activities. It will cover the basics of prototyping – the benefits, some case study examples and a range of approaches to consider. Practical activities will involve iterating the design of a simple interface; first with paper and then with the online prototyping tool Balsamiq.

Registration:

For the FULL DAY including practical workshop register here: http://www.nesc.ac.uk/esi/events/1196/

If the full day event books up or you only want to attend the morning presentations, register here: http://www.nesc.ac.uk/esi/events/1199/

For more information including details on the instructors visit our project Wiki: https://www.wiki.ed.ac.uk/display/UX2/UCD+Day

Introduction

Following on from the usability testing of the desktop prototype digital library, we conducted more user research on the mobile prototype. The prototype is similar to the full version but with some services removed to create a simpler interface. The facet navigation situated in the right column is now provided within a ‘Refine’ link. The ability to bookmark items is also available. As before, the prototype is based on an open source ruby-on rails discovery interface, Blacklight which has been further developed throughout the project. The prototype indexes the catalogues provided by the National e-Science Centre at The University of Edinburgh (NeSC) and CERN – The European Organisation for Nuclear Research.

The data capture and test methods are detailed here alongside the main findings.

Method

On the 10th and 11th March 2011, usability testing was conducted on the UX2 mobile optimised digital library prototype with six selected students from the University of Edinburgh (UoE).  Each test was carried out as a one-to-one session and comprised an explanation of the research, undertaking task-based scenarios followed by a short post-test interview and word-choice questionnaire. A full description of the prototype and the changes made to the desktop version to optimise it for mobile devices will be documented shortly on the associate project blog, Enhancing User Interactions in Digital Libraries.

View of mobile device from webcam setup

Six participants were recruited from the same list which was originally compiled for the focus group recruitment in January. Only those who owned an iPhone were invited to take part as this was the platform the prototype had been optimised for. Each participant was given a £10 Amazon voucher as payment for their time. Each session lasted between 50 and 65 minutes. A few additional statistics on the participant’s profiles are provided below:

  • All 6 participants owned an Apple device (iPhone 3GS, iPhone 4 or iPod touch)
  • 3 participants had attended our focus group in January, 3 had not
  • 4 participants were undergraduates, 2 were postgraduates
  • 50:50 male to female ratio
  • All were on a pay monthly contract for their mobile phone device

Complete setup with webcam positioned on perspex above phone. Also the additional webcam to capture body language.

To record the testing we used a similar set up to that suggested by Harry Brignull on his blog. It was fairly low-cost (approx £80 excluding Morae software) and required two webcams, a small piece of perspex (approx 35cm x 10cm), testing software (we used Morae v3 as it allowed us to capture from 2 webcams), cheap plastic mobile phone cases for each type of handset being tested (two in this case) and velcro to attach them to the perspex. It was very easy to mould the perspex to shape by heating it gently above something readily available, a toaster! The Logitech C905 webcam we used had a great advanced settings which allowed us to mirror and flip the image, making it easy to decipher what was going on (see image). Overall the setup worked well as it was lightweight and relatively unobtrusive. The camera was positioned in the best place at all times and this allowed us to see exactly what was going on as well as record participant’s interactions and body language using a second webcam.

Task scenarios

The scenarios used were adapted from those used for the desktop usability tests. They were designed to test a variety of the prototype’s features. The main focus was the facet navigation system (Refine link), the presentation of information in the item detail page and the bookmarking service:

  1. As part of your coursework your lecturer has asked you to read a recent presentation on fusion. Using the prototype can you find a suitable presentation published in the last 2-3 years.
  2. You have to write an essay on quantum mechanics, can you use the prototype to find several resources which will help you get started? If you wanted to save these items to refer to later when you’re at a computer, how would you do this?
  3. You are looking for a paper on Grid Computing to read on the bus. Can you use the prototype to find a suitable digital resource?
  4. You are looking for a presentation given by Harald Bath at the EGEE summer school; can you find the video or audio presentation to watch at the gym?

Word Choice Results

The aim of the word choice exercise was to elicit an overall impression of the participant’s experience using the prototype. The word choice exercise presented participants with a list of 97 adjectives (both positive and negative) at the end of the session and asked them to pick those they felt were descriptive of the prototype

Word cloud results

Each of the words shown in image above represents a word that was ticked at least once. The larger and darker the word is, the more often it was ticked by participants.

The most prominent words were all positive: Accessible, Straightforward, Easy to use, Clean and Useful. These words were chosen by 5 of the 6 participants surveyed. Other positive words include Clear, Convenient, Effective, Efficient, Flexible, Relevant and Usable. Some of these words are a particularly interesting as they could directly relate to the ability to access the service anywhere using a mobile device.

Some of the negative words selected include: Simplistic (could also be positive), Ordinary, Ambiguous, Inadequate, Ineffective, Old and Confusing. However some of these words were used by a minority of participants and were therefore not as visible. The words are not all that surprising considering the mobile website is a prototype with fewer services than the desktop version.

Findings

Finding Information (Navigation and refine)

Some students noticed that there was no universal ‘Home’ link throughout the site. Students felt that a shortcut back to the home page would be a good idea, despite the fact that the search field was present on the results page. This suggests a desire from students to be able to skip pages instead of navigating back through pages one at a time.

NCSU Mobile search form

Many of the students stated a need to have an advanced search facility. Often students were looking for either an Advanced search link near the search field or a drop-down menu where they could specify details such as Author, Title, Subject/Keyword. As this is a feature they encounter in other search services, they still expect to have access to it on a mobile device. An example of a mobile library site providing such a service is North Carolina State University (see image).

The desire for an advanced search was  problematic for users who did not see the ‘Refine’ link at the top of the page. Those users struggled to complete tasks and consequently their experience of the prototype was affected. Failing to use the Refine service made using the prototype much more difficult. This was demonstrated when students were asked to search for items from a specific year (task 1). Students had no way of knowing if they had searched through every relevant result without looking at every page of results. This finding suggests that it is imperative that a solution to the Refine service be provided. When the Refine link was pointed out to students they stated that they did not see the link due to its location at the top of the screen or did not fully understand its label. These students were searching for an ‘Advanced’ label located close to the search form and search results.

Those students who did use the refine service wanted to filter by more than one item at a time. This was particularly evident in task 1 when students were searching for presentations published in the last 2-3 years. Students found it quite laborious to select one year at a time to review the results. They wanted to be able to search a period of time, either the last 5 years or set the search terms themselves (possibly using a form or sliders). This finding was also revealed during the desktop prototype testing. It demonstrates that this is an important criteria for students to make searching easier and remains so when using the service on a mobile device.

The refine service itself was generally well received when used. The facets provided were considered useful to students and listed in a logical order of importance. One student realised that the items listed in the year facet was not necessarily the last ten years, rather the top ten results. This caused some confusion as first and could affect the success of student’s tasks. In addition, students found using the refine service laborious at times with unnecessary steps involved which could prove troublesome on a mobile device. An idea of the task flow is provided below to outline the steps involved in a typical task using the refine service:

Select ‘Refine’ > Select ‘2010’ from Year facet > [view results] > Go back to ‘Refine’ > Remove 2010 > [view results] > Go back to ‘Refine’ > Select ‘2009’…

or

Select ‘Refine’ > Select ‘2010’ >[view results] > Remove 2010 from results page > [view results] > Go back to ‘Refine’ > Select ‘2009’…

Note: actions in square brackets happen automatically in the process.

Several students used the first system to remove facets perhaps because they did not notice the option to do this on the results page. Consequently, the additional number of steps involved appeared to affect the user’s experience. This was also one of the reasons that students wanted to be able to select a range of dates at once instead of going backwards and forwards within the refine service. One student suggested a tick box option where they could select all the items within each facet they wanted to use to narrow their results at once.

Saving Information (Bookmarking service)

When students were asked to save information to refer to later as part of task 2, some used the bookmarking service. Those who did found it relatively easy to bookmark an item and retrieve it afterwards. The feedback seemed to suggest that it was clear when an item had been saved and the folder icon at the top of the screen was clearly visible. However, there seems to be a bug when a user tries to bookmark an item before logging in. Upon selecting the ‘Bookmark this’ link and completing the login form, users are taken to their bookmarks page without the item listed. The user has to go back to the results page and attempt the action again for it to be successful. In addition, although users could see the number of items bookmarked next to the folder icon, it might be more difficult to spot under different lighting conditions. The white text on pale green background makes less likely to stand out.

There was also a desire for additional features within the bookmarking service which would make it a more effective tool. Additional information on each item including author, date, location and most importantly, shelfmark, would make it easier for users to distinguish items with the same title and locate a number of saved items in the library quickly. Students also wanted to be able to categorise bookmarks into sections, similar to the folder system in a browser’s bookmarking service. Being able to export bookmarks by emailing them to an address was also considered a useful feature to provide.

Although the usability of the bookmarking service was considered to be good, the usefulness was not apparent to every student. Many students have their own system in place for recording items of interest in the library. Writing details down on paper, copy & pasting text to a separate document or simply minimising the browser to open again at the relevant moment. The Safari browser on iPhones allow users to have several windows open at once and automatically saves the state of all windows when it is closed. One participant in particular used this system to park information in order to retrieve it when required. Consequently they did not envisage themself using the bookmarking system. It is interesting to note that of those students who participated in the study, none of them stated that they used the existing bookmarking system available in the University’s own library website called ‘My List (Bookbag)’ .

Reviewing Information

Although the UX2 library is a prototype, students felt that the level of information provided on each item was for the most part adequate. Being able to view documents was often expected by students but was not always possible. One student questioned whether everyone would be able to preview documents if they did not have a Google account as this is requested upon selecting the electronic document. This could be problematic, especially because feedback from the focus groups indicated averseness to downloading files to mobile devices due to limited data storage.

In addition to previewing documents, links to information sources under the heading ‘Links, Files’ was not easily understood by students. The links were not easy to identify because they are not presented as conventional blue, underlined text. The long URLs and small text size also made it difficult for students to guess where the links might lead. In many situations, the links did not meet user expectations. Students would select a link expecting it to lead them to the full text item when instead it went to an external website. Students wanted to be warned when links lead to an external (and often not optimised for mobile) website. Although this was an issue during the desktop usability tests, it became even more apparent among students using the prototype on a smaller screen.

Something which was requested by students not only during the usability tests but also in the focus groups was the ability to easily access a map of libraries. Having such information would make it much easier to locate books, particularly when students are not familiar with the particular library. Students felt a link to such a map could be provided on the item information page and located under the library holding data to make it easy to access. There was also a desire to provide a simple system which informs students when an item is available or on loan. A colour coding system or simple icon was suggested which could be displayed in the results page next to each item – green for available, red for on loan. A library which has gone some way to address this need is NCSU. They provide users with the opportunity to filter out items which are not available (see earlier image).

Post test interview

Overall students felt that the prototype was fairly effective in helping them find information. Those who did not see the ‘Refine’ link naturally believed it could have been clearer. Another student stated that quality of results was sometimes an issue, suggesting the need for improvement. The timeliness of resources was often dependent on the subject students were studying. Subjects where research tends to move quickly, such as technology, recent publications are much more important. Students were asked to state which two things which they particularly liked about the prototype. Their answers are listed below:

  • Refine page (2)
  • Item information page
  • Level of information provided – not overloaded
  • Font style is modern
  • Minimalist and contemporary design (2)
  • Simple search
  • Bookmarking system (2)
  • Being able to filter results by type e.g. book, presentation

Some things that students believed could be improved:

  • Refine search – visibility and task flow
  • Design, particularly the home page – no logo, name or clear description of purpose (2)
  • Provide a link to home page throughout the site (2)
  • Provide search options next to search form
  • Improve the date range of the Date facet
  • Tips to guide you through the website
  • Visibility and source of links on item page
  • Provide an additional option to search/narrow results by library

Conclusion

Observation of the usability tests showed that participants coped well undertaking tasks using a smaller screen. The biggest issue was the visibility of the refine page which contained the facet navigation service. When participants were not aware of this option, their experience of using the prototype was severely compromised. Those who did use the refine service were able to complete tasks more efficiently but did find the number of steps involved to do so unnecessary. This suggests further work is required on the implementation of a facet navigation service to improve its usefulness and usability. Although some of the students appreciated the minimalist nature of the prototype, there was still desire to undertake more than just simple searches. The bookmarking service was on the whole well received and was considered useful with the addition of a few more features. However, the uptake of such a service is still unknown as students often had existing bookmarking systems in place.

In November we ran a survey to all students at the university to investigate mobile demographics among students, Internet habits and attitudes towards mobile library services. The survey ran for two weeks and in that time 1,716 students responded. This was helped by an incentive prize draw of £50 Amazon vouchers – very attractive at this time of year.

In addition to demographic data and data on student’s mobile device, students were also asked about their mobile Internet habits and specifically their exposure to mobile university and library services. Finally they were asked which of the proposed mobile library services they would find most useful. The survey was loosely based on the research conducted by Information Services (IS) in March 2010 on mobile university services. Doing so allowed some comparisons to be made while also investigating mobile library services in-depth. Findings from the IS Survey can be found here: http://www.projects.ed.ac.uk/areas/itservices/integrated/ITS045/Other_documents/MobileSurvey2010.shtml.

The survey is part of the wider research within the UX2.0 project and relates to Objective 3 (deliverable 5.3) which is to evaluate the user experience in specific contexts involving real user communities. The quantitative data gathered from the survey will be supplemented by the focus group planned at the start of next year. The findings from the survey will help to shape the direction of the focus group which in turn will hopefully support the findings from the survey. Check back for a report on the focus group in the near future.

Some of the headline findings are provided below. The full report can be viewed here: http://bit.ly/ux2mobilesurvey.

  • 66.5% of students surveyed own a smartphone. This is an increase of 17.3% from the IS survey in March 2010.
  • Apple iPhones accounted for 21.9% of smart handsets, followed closely by Nokia at 20% and Samsung at 15.3%.
  • 68% of students have pay monthly contracts.
  • 74% students have either a contract that provides unlimited access to the Internet or provides sufficient access to meet their needs.
  • Services which students access online most frequently (several times a day or daily) are websites in general, email and social networks.
  • Activities which students are least likely to carry out on their mobile handsets regularly included downloading content and uploading images to photo sharing networks.
  • The biggest frustrations students experience using the Internet on mobile handsets included slow or limited Internet connection speeds, poorly designed mobile websites or websites without a mobile compatible website and the limitations of using a small screen with small buttons.
  • The highest proportion of students surveyed stated that they had not tried to access library services using their mobile device.
  • The top 3 potential University library services which students would find most useful would be:
    • Search library catalogue
    • Reserve items on loan
    • View your library record – see your charges summary and which books are reserved, requested, booked and loan

It was interesting to find that over half of students surveyed had not tried to access library services on their mobile device. It does seem that students use their mobile for other University services such as student email, the MyEd student portal and Web CT as well as other general university information such as shuttle bus timetables. Perhaps a mobile optimised library website with useful and easy accessible information would encourage students to use library services on the move more often. That coupled with effective communication that these services exist for mobile devices.

Students do seem open to the idea of mobile library services being useful, with the most useful being access to library records, the ability to search the library catalogue and databases, reserve items on loan and locate a shelf mark. However, the usefulness is dependent on the implementation of such services as many students reported that websites not optimised for smaller screens can be very difficult to use.

Limitations of smaller screen:

“It is slower than a computer and the screen is too small for a full-sized website. The library should have a special mobile website which does not take long to load and is easy to navigate on a small screen.”

Implementation of mobile websites:

“Websites should be tailored in order to be suitable for rendering on small screens. Endless scrolling makes for bad usability.”

Percentage of students who rate each library mobile service as “Very useful” or “Generally useful”

Rank % Library Service
1. 93 View your library record – see your charges summary and which books are reserved, requested, booked and loaned
2. 92.5 Search library catalogue
3. 90 Reserve items on loan
4. 89 Search library databases
5. 87 Locate a shelf mark
6. 84.5 Check PC availability in library
7. 82 Request an item through inter-library loan
8. 71 View maps of libraries
9. 67 Receive alerts relating to library information or services etc.
10. 64 Library Maps & Locations using GPS – find your way around University libraries
11. 58 Library statistics i.e. top books in different categories and popular searches
12. 55 Friend Locator – see where friends are in the library and contact them to meet up
13. 54 Read reviews others have left on items in the library
14. 52 Share items that you’ve found and/or read in the library that you think others will find useful
15. 49 Rate and review items from the library

In order to get a picture of the mobile library landscape, the mobile offering from a variety of academic institutions has been investigated. The aim of this is to understand what institutional libraries are currently doing to provide mobile services and identifying those services which The University of Edinburgh could benefit from.

North Carolina State University

NCSU have created a mobile library website which looks similar to a mobile application. The design is very similar to smart phone interfaces and this gives the website a modern look. Services which the library provide in addition to the catalogue include room booking, an instant messaging chat system with librarians and PC availability. Interestingly they also provide more unconventional services such as group finder and live webcam feeds. The group finder section helps users locate study groups in the library and the live webcams make it possible from users to monitor locations of the library remotely, including the cafe. It has been reported that the webcams generated over a third of mobile page views in the first 8 weeks that the mobile services were introduced suggesting that the webcams can be very useful to users (Meier, 2010).

Amsterdam University Library

Driek Heesakkers recently presented the case study of Amsterdam’s mobile library strategy during CILIP’s Executive briefing day, Becoming Upwardly Mobile – can libraries rise to the challenge? Amsterdam University Library‘s short-term mobile strategy resulted in a mobile web presence from a budget of €1500. This interim mobile library website provides a number of services including access to the OPAC, a simple search interface, a link to the library’s social networking page on Twitter, information on library opening hours, location, contact details and PC availability throughout the library. In addition to these basic services, the catalogue provides a simplified advanced search feature with a drop-down menu and access to the user’s library account. The interface of the search is simple with basic information in search results.

Huddersfield University and University of Bath

Huddersfield University have taken another approach to providing services on mobile devices. Using QR codes in and around their library they are able to provide links to digital information. This includes digital copies of print journals, links to mobile friendly videos and other useful digital resources. The QR codes appear at the end of book shelves, signage and on printed hand-outs. They even provide a QR code with a video on using the print credit machine (see image below). Although this technology may not be well-known among students it does have the potential to bring useful resources to students through their mobiles. As smartphones become more common place, the barrier to this technology is likely to be significantly lowered.

The University of Bath have also been using QR codes in their library. Their project explores the idea of using QR codes to help users locate items more easily by launching a map of the library on user’s mobiles. They also carried out a cross-institutional study which revealed that students are becoming more aware of QR codes and find the idea of using QR codes very appealing.

Image credit: Andrew Walsh, University of Huddersfield

Cambridge University

Cambridge University is one of the other UK university’s mentioned on M-Libraries wiki that have developed a mobile friendly site of their digital library. The mobile site focusses on allowing users to search the catalogue and access their account on the move. Notably they also highlight the benefits of logging in to the site for extra benefits including renewing items on loan, make and manage requests and build a list of items that can be emailed to any address. This would be particularly useful for research students who might otherwise write down their search results onto paper. The catalogue search is fairly simple with the addition of thumbnail images of book covers where relevant. Other useful features which are provided in the item details include a link to the Google Books preview of the full text and a map highlighting where the shelf mark is located within the library. Clearly some thought has been made to consider what information user’s need on their mobile when designing the site.

A comparitive analysis of  library mobile services in use in higher education was conducted by California Digital Library. They found that mobile library services fell into five categories 1. Library mobile website; 2. text messaging services; 3. mobile catalogue search; 4. access to resources, and 5. new tools and services. Simple information such as hours, direction, map, news and floor plans are common in many mobile library websites. Jeff Penka from OCLC supports this idea when he commented at CILIP’s Executive briefing day that “The library catalogue isn’t the only service (users) want to access via a mobile, they want to find out which computers are free, paying fines, deal with reservations, see opening hours too.”

Catalogue search is also a popular mobile service which is provided by academic digital libraries. This supports the findings from a survey conducted by Cambridge University which found that 55% of respondents were in favour of bring able to access the library catalogue from a mobile phone (Mills, 2009).

A list of resources relevant to mobile library services have been collected and compiled into a Delicious links folder. These links are open to everyone and I encourage you to add other resources I may have missed. Thanks

http://www.delicious.com/lorraine_p/bundle:UX2.0%20Mobile%20Research


Last month a usability study was carried out on the UX2 digital library prototype. The study involved 10 student participants who tried to complete a variety of tasks using the prototype. The report is now available to read in full and can be accessed via the library (http://bit.ly/ux2usabilityreport1).

The prototype is based on an open source ruby-on rails discovery interface, Blacklight which has been further developed for the project to provide additional features. Existing component services have been ‘mashed-up’ to generate the UX2.0 digital library. The prototype currently indexes the catalogues provided by the National e-Science Centre at The University of Edinburgh (NeSC) and CERN – The European Organisation for Nuclear Research. The report presents the findings of the usability testing (work package 2 – WP2.3) of the prototype which was conducted with representative users at the university. The study reveals a range of issues uncovered by observing participants using the prototype by trying to complete tasks. The findings outlined in the report provide a number of recommendations for changes to the prototype in order to improve the user’s experience.

In order to identify and fully explain the technology responsible for each issue in the report, supplementary blogs will be published on the project website in stages, as and when developmental changes are made (follow the ux2 Twitter account for announcements). It is hoped that this supplemental development documentation will make it more accessible to other digital library developers and the wider JISC community. Some of the main findings from the report are summarised below.

Positive findings from the study highlighted positive aspects of the prototype:

  • Allowing users to narrow results using the faceted navigation links was useful.
  • Providing users with details of the item content including full text preview, video and presentation slides was informative.
  • Allowing users to bookmark items and add notes was considered a useful feature.
  • Overall the layout and design was clean, simple and useful.

These positive findings from the testing are reflected in the word cloud questionnaire participants were asked to complete:

UX2 word cloud

However there were some usability issues with the prototype:

  • It was not obvious when the system saved previously selected facets in the Scope, often misleading participant’s expectations.
  • The external ‘Other’ links were not relevant to participants and often mistrusted or considered a distraction.
  • It was not clear when an item had a full text preview feature
  • Links to information resources were not prominent and often missed by participants
  • The format of text within the item details page made it difficult to read and consequently participants often ignored it.

There were also a few lessons learned from the user study which I thought would be useful to share:

  1. Recruiting participants via telephone does not necessarily guarantee attendance. Two participants did not show up to their slot after arranging the appointment with them by phone and sending them an email confirmation. However, this could also have been affected by the time of year. It transpired that many students had coursework deadlines the same week and the offending students did say they forgot because they had a heavy workload.
  2. User generated tasks are not easy to replicate using prototypes. This was not unexpected but something which was tried anyway. As suspected, it was difficult to create a task which could generate results when using such a specialised and relatively small database. However, when it was successful it did return some useful findings.
  3. It’s difficult to facilitate usability tests and log details using Morae. Any usability practitioner will tell you that it’s important concentrate on observing the participant and interacting with them and to avoid breaking the flow by stopping to take detailed notes. I found it impossible to observe a participant, engage with what they were doing and log behaviour on Morae so would recommend you recruit a note-taker if this is important for your usability study.

Last week during our regular project meetings we decided on a roadmap for the user research aspect for the last few months of the project (is it that time already?!). In the spirit of transparency I decided to publish my plan here. I’ll of course be blogging along the way, reporting findings and evaluating the work as I go. If you can point me to any existing research that I should include in my investigation, please feel free to leave a comment with a link at the end and thanks in advance!

Background

This research will examine the current use of Web 2.0 developments in digital libraries, their use scenarios and applications. The scope of the remaining research will be centred on services which diffuse the digital library through mobile technology. The aim of this mobile research is to identify areas that may be relevant to The University of Edinburgh (UoE) and evaluate a prototype mobile library service.

Project Aim

To enhance the user experience of digital libraries through technological developments centred on usability inspection and evaluation.

Project Objectives

  1. To undertake usability inspection and contemporary UX techniques research
  2. To enhance digital libraries with state-of-the-art technologies
  3. To evaluate user experience in specific contexts involving real user communities

Mobile Research Aims:

1. Review current mobile digital library landscape, how services are diffused using mobile platforms and what UoE can learn (Obj3)

Formative evaluation informed by existing mobile digital library services and usability studies of mobile library services will be undertaken. This will help to provide a clear picture of the mobile digital library landscape which will inform the project’s own development work. Existing mobile usability research[i] will also provide insights into existing user centred design processes which can be adapted for this project.

Output: Blog series reviewing the trends in mobile digital library services, highlighting successful services and identifying what UoE can learn from other projects. In addition, a list of existing mobile digital library resources will be created as a resource for others.

2. Review good UX practices for mobile applications & websites as well as usability evaluation techniques (Obj1)

As mobile usability is a relatively new subject to the project, research will be conducted on usability practices for mobile design and development.  In addition, mobile evaluation methodologies will be identified and incorporated into the prototype evaluation (Obj2).

Output: Blog post which highlights good mobile UX resources and describes the evaluation technique which will be applied to the project.

3. Investigate what users want from a mobile library service (Obj3)

Continuing on from the Mobile Services survey conducted by Information Services (IS) in March 2010[ii], a subsequent survey will be conducted with UoE students which will focus on mobile library services. The findings will provide insight into the types of services users would find useful and this will hopefully influence the direction of development. The research will also help to support the ongoing mobile services development by IS as it will provide additional data which can be benchmarked against their previous survey.

The quantitative data gathered from the survey will be supplemented with a focus group with those likely to be using the service (end users) and those helping to provide the service (staff, developers, librarians). The findings will not only help to qualify the findings from the survey but also provide a broad perspective of how a digital library service should be shaped by including all stakeholders.

Output: Survey report detailing findings and outcomes from first focus group with stakeholders.

4. Evaluate the usability of the prototype mobile library service (Obj1)

The usability of the higher fidelity prototype will be evaluated with representative users. These one-on-one sessions will take place with a small number of users (6-12) and will be conducted using a simulator, a smart phone or the user’s own mobile handset. Qualitative date will be captured and reported. The objective of this usability study is to ensure the success of the prototype and provide a use case for digital library services at the University of Edinburgh and beyond.

Output: Summary of findings from focus group and detailed usability test report.


Now that the usability testing has been concluded, it seemed an appropriate time to evaluate our recruitment process and reflect on what we learned. Hopefully this will provide useful pointers to anyone looking to recruit for their own usability study.

Recruiting personas

As stated in the AquabrowserUX project proposal (Objective 3), the personas that were developed would help in recruiting representative users for the usability tests. Having learned some lessons from the persona interview recruitment, I made a few changes to the screener and added a some new questions. The screener questions can be seen below. The main changes included additional digital services consulted when seeking information such as Google|Google Books|Google Scholar|Wikipedia|National Library of Scotland and an open question asking students to describe how they search for information such as books or journals online. The additional options reflected the wider range of services students consult as part of their study. The persona interviews demonstrated that these are not limited to university services. The open question had two purposes; firstly it was able to collect valuable details from students in their own words which helped to identify which persona or personas the participant fitted. Secondly it went some way to revealing how good the participant’s written English was and potentially how talkative they are likely to be in the session. Although this is no substitute for telephone screening, it certainly helped and we found that every participant we recruited was able to talk comfortably during the test. As recruitment was being done by myself and not outsourced to a 3rd person, this seemed the easiest solution at the time.

When recruiting personas the main things I was looking for was the user’s information seeking behaviour and habits. I wanted to know what users typically do when looking for information online and the services they habitually use to help. The questions in the screener were designed to identify these things while also differentiate respondents into one type of (but not always exclusive) persona.

Screener Questions

The user research will be taking place over a number of dates. Please specify all the dates you will be available if selected to take part

26th August |27th August | 13th September | 14th September

What do you do at the university?

Undergraduate 1st |2nd |3rd | 4th | 5th year| Masters/ Post-graduate | PhD

What is your program of study?

What of the following online services do you use when searching for information and roughly how many hours a week do you spend on each?

Classic catalogue | Aquabrowser catalogue | Searcher | E-journals | My Ed | Pub Med | Web of Knowledge/Science | National Library of Scotland | Google Books | Google Scholar | Google | Wikipedia

How many hours a week do you spend using them?

Never|1-3 hours|4-10 hours|More than 10 hours

How much time per week do you spend in any of Edinburgh University libraries?

Never|1-3 hours|4-10 hours|More than 10 hours

Tell me about the way you search for information such as books or journals online?

Things we learned

There were a number of things that we would recommend to do when recruiting participants which I’ve listed below:

  1. Finalise recruitment by telephone, not email. Not surprisingly, I found that it’s better to finalise recruitment by telephone once you have received a completed screener. It is quicker to recruit this way as you can determine a suitable slot and confirm a participant’s attendance within a few minutes rather than waiting days for a confirmation email. It also provides insight into how comfortable the respondent is when speaking to a stranger which will affect the success of your testing.
  2. Screen out anyone with a psychology background. It is something of an accepted norm amongst professional recruitment agencies but something which I forgot to include in the screener. In the end I only recruited one PhD student with a Masters in psychology, so did not prove much of a problem in this study. Often these individuals do not carry out tasks in the way they would normally do, instead examining the task and often trying to beat it. This invariably can provide inaccurate results which aren’t always useful.
  3. Beware of participants who only want to participate to get the incentive. They will often answer the screener questions in a way they think will ensure selection and not honestly. We had one respondent who stated that they used every website listed more than 10 hours a week (the maximum value provided). It immediately raised flags and consequently that person was not recruited.
  4. Be prepared for the odd wrong answer. On occasion, we found out during the session that something the participant said they had used in the past they hadn’t seen before and vice versa. This was particularly tricky because often students aren’t aware of Aquabrowser by name and are therefore unable to accurately describe their use of it.

Useful resources

For more information on recruiting better research participants check out the article by Jim Ross on UX Matters: http://www.uxmatters.com/mt/archives/2010/07/recruiting-better-research-participants.php. There is also a similarly useful article by Abhay Rautela on Cone Trees with tips on conducting your own DIY recruitment for usability testing: http://www.conetrees.com/2009/02/articles/tips-for-effective-diy-participant-recruitment-for-usability-testing/.

Have I missed anything? If there is something I’ve not covered here please feel free to leave a comment and I’ll make sure I respond. Thanks

In this final part of the persona creation series I will introduce the personas we created and discuss how we plan to keep the personas relevant and current. As mentioned in the previous blog, we created 4 personas based on our interviews. These personas were:

Pete, the progressive browser
Badaal, the search butterfly
Eve, the e-book reader
Sandra, the search specialist

Full details of each persona can now be viewed on the project wiki. It is also possible to download a Word version of the personas from the Wiki if required.

Looking at the full personas in the wiki, there a few features of them worth mentioning:

  • We decided to recreate some of the scales we used when segmenting the groups. We felt that this provided a quick snapshot of the persona in addition to the more descriptive background which hopefully brought the persona to life.
  • We used photos from iStockphoto.com and http://www.sxc.hu/
  • In addition to background and demographic information we also added sections including Personal Goals, Frustration & Pain Points, Typical Tasks and Information Seeking Behaviour. These were based on the categories we used when writing summaries for each participant.
  • We tried to create an alliteration of the persona’s name and their characteristic behaviour to make them more memorable e.g. Pete the progressive browser

Dissemination and future-proofing personas

In addition to publicising the personas through the blogging streams (WordPress and twitter), we have tried where possible to include as much raw data and project documentation as possible on the project wiki. As mentioned in part 1, we had difficulty recruiting staff and consequently were unable to create a persona which represented them. If it’s possible to conduct interviews with University staff members in future projects, an additional persona for staff could be added to the collection. By providing as much raw data as possible in addition to full explanations of the methods used, it should be possible for other groups to create their own personas using the same template.

During this year’s UPA Conference in Munich a presentation on mobile personas highlighted the importance of making them forward thinking. The presentation discussed the dangers of personas becoming outdated and the consequences of this on product design. This is not just important in mobile design, outdated personas are potentially dangerous and could have a negative rather than positive impact on a project. Tom Allison presents a number of ways that personas can become outdated or ‘Zombies’ in his excellent presentation, UX in the Real World: There’s no such thing as “No Persona” (see descriptions below). Reflecting on the process we have taken to create our personas, I am confident that our personas are not zombies, however that is not to say that they cannot ‘be turned’ in the future. To guard against this possibility it is important to encourage others to continue the work, adding more interviews and details appropriate to their work in order to make them appropriate. In our project we were interested in the persona’s attitude to UoE Aquabrowser and Voyager catalogues. However, the information seeking behaviour of each persona is much more general and has the potential to be used as a basis for other academic library use-cases. Providing a thorough account of how the personas were created should hopefully make it easier for others to create their own set of system-specific personas.

Descriptions of the different types of Zombie Personas by Tom Allison:

Mirror Personas: These are the end-user models that get used in the absence of any other reference or description. Anyone on the project team that needs to make a design decision simply asks themselves what they would want in that situation (i.e., they metaphorically look in the mirror). Usually that is not a very good reflection of what the targeted end-user would actually want. Often the difference in these two perspectives can render a result anywhere from terribly frustrating to completely useless to the eventual end-users.

Undead Personas: These are personas that were in fact constructed at one point in a project but that are no longer truly “alive” to the project. They may be hanging on the design team’s wall, or sitting in a report somewhere online or in a drawer. They exist (that’s the “undead” part), but they exert none of the positive effects that a “good persona” can. They may be the worst sort of persona in that they give the team the false confidence that they have “done personas” and they are probably most responsible for the bad reputation that personas sometimes have – in that, the team “did personas” but “it was a waste of time” because they did not keep them alive to the overall process in order to reap the benefits of their work.

Unicorn/Frankenstein Personas: These are personas dreamed up or slapped together from pre-existing parts by someone in the project team. Regardless of how they are used, the signal they give is not a true one. They no more reflect the actual end-user than the programmers mirror persona does and team members who understand that they are based on nothing more than someone’s imagination tend to resent them rather than view them as a resource. The don’t work like good personas and they are resented by those in the know. These along with their undead brethren lead to many with limited experience with personas to have a negative impression of them.

Stupid User Personas: These are perhaps the hardiest of the zombie personas. If personas are built or even just thought about and kept out of the “light of collaborative day” – that is, are not shared and publicized widely and integrated into every stage of a process – then they tend toward negative or dismissive  models of the end-user. Teams whose only access to the end-user is a combination of direct communication, interruption and negative feedback to their delivered product are very likely to cultivate these personas in the absence of “good personas”.

Conclusions and reflections

It’s difficult  to be subjective when evaluating the success of personas you were involved in creating, particularly because the personas have only just been created. Feedback from others will hopefully provide one way to measure their success. The realism and believability of the personas is important and something I believe we have managed to achieve but I am always interested to know if others agree and if there are any improvements we could make. Having spent a little time working with some* of the personas to recruit participants for the usability testing of Work package 3 (Objective 3) I have made some observations:

There are not huge differences between Baadal and Eve. The differentiating factors are the consumption of digital resources (e.g e-books) and use of the University digital libraries. On many other scales they are the same or very similar. This has made it a little trickier to recruit to each persona as often the details provided by willing participants makes them difficult to categorise as one or the other. This evidence provides a compelling argument that these personas could be merged. However, there is also an argument that participants do not have to fit neatly into one persona or another. For the purpose of the project we will continue to use three individual personas. It will be possible to evaluate the success of the personas over the course of time.

* Findings from the interviews led to the decision to test with students as these are the primary target audience for UoE Aquabrowser. Consequently we used personas Pete, Badaal and Eve in recruitment.


del.icio.us bookmarks

Twitter feed

Archive