Researching Usability

While researching mobile library services, I’ve read a few research papers produced by other projects recently. Doing so has provided some guidance on our own research techniques and allow me to make comparisons with our own findings. One very comprehensive project in particular has proved quite informative: The California Digital Library Mobile Device User Research Project conducted a range of thorough research on the mobile strategy for their institution. Their Mobile Strategy Report includes details of surveys and interviews which they carried out earlier this year. The findings were used to detail barriers to mobile use and make recommendations when designing for mobile devices. I’ve decided to present some of their findings as I have found many similarities to our own findings. The design recommendations are also useful as they apply to all higher education institutions who may be thinking about developing mobile services. I do recommend you read the full report if you are interested in this area of research and design.

Designing Native Applications and Mobile Websites

There is a lot of debate on whether to design for native applications or provide a mobile website. As native applications are designed for a particular platform they work on a limited number of handsets. It also means that you may have to design a native applications for each platform separately which could be three or more if required. However, a mobile website is accessible to more mobile users as users are not required to download anything before use and you don’t need the approval of an app store like iTunes before it is available. California Digital Library (CDL) research indicates that a mobile website is currently the best option. Perhaps the website should be prioritized first and then development for a native application can follow. The report points to the book by Brian Fling, Mobile Device Design and Development which provides guidelines on the circumstances when a native application is the best route:

  • Charging for it
  • Creating a game
  • Using specific locations (though some devices are able to detect location through browser applications)
  • Using cameras
  • Using accelerometers (to detect motion or rotation)
  • Accessing file systems
  • Offline users

Research findings and recommendations:

Smartphone ownership: “Smartphone ownership will likely increase over the next couple of years.” – the findings from the surveys conducted at University of Edinburgh in March and November this year support this conclusion as they have shown an increase in smartphone ownership by 17% to 67%.

Activities on mobile devices with internet: “They are used less for academic purposes and are also not used often for uploading content.” – this is also supported by our own survey findings which revealed a similar story. Top activities on internet capable mobile devices included finding information such as directions or opening hours, email and text messaging, reading news articles and blogs but not academic content, social networking and video & music consumption.

Library resources for research on internet enabled mobile devices: “Most interviewees did not want to do actual academic research on their mobile devices. Despite this there was still interest in having the option to access library databases, catalogues and resources from mobile devices.” A great metaphor is referenced in the report which describes Internet consumption on mobile devices very well:

Desktop internet is like scuba diving, where the search can be “immersive” and “invites exploration and discovery.” Mobile internet use is closer to snorkeling, where”shallow dipping in and dipping out of content for quick checking of key content is desired.

(Hinman et al., 2008)

– Again this supports our findings which showed that half of respondents had not accessed any library services using their mobile devices.

CDL Recommendations:

  • Keep in mind that ownership of internet capable mobile devices may not translate to high usage of academic mobile tools.
  • Gear mobile sites and apps towards the mobile user who is looking for quick pieces of information (no complicated tasks).
  • Make library websites, databases and catalogues available on mobile devices, but do not provide all the functionality of desktop versions.
  • Provide easy methods to transfer research and resources to other devices, such as by email.

Reaching your mobile audience: Something that was not directly surveyed at UoE, California Digital Library found that 60% of students ranked fellow students as the most likely sources for finding out about new tools and services. Consequently spreading the word about new tools and services can be difficult. This is something for UX2 and UoE to be aware of when designing mobile library services.

CDL Recommendations:

  • Auto-detect mobile devices and automatically display mobile version of a site (though always provide obvious links back to the full version of the site).
  • Create mobile websites rather than apps whenever possible so that users do not  need to download software in advance of using it.
  • Advertise through a variety of channels, including campus or departmental emails, campus websites, library websites, and blogs.

Overall CDL design recommendations:

  • Set up testing practices and environments for the most heavily used device platforms (Apple iOS,  RIM Blackberry OS, and Android OS). Test with physical devices where possible rather than emulation environments.
  • Support mobile web access as opposed to building standalone mobile applications.
  • Adopt and maintain web analytical tools to accurate mobile device tracking and usage statistics for online services.
  • Continue to survey constituents and end-users annually or bi-annually to capture rapidly changing behaviours.

The recommendations provided are based on the research findings and are therefore very useful. Evidence supporting the creation of a mobile website first is something which we hope to explore in the focus group next year. As other platforms follow Apple’s example and provide native applications, it will be interesting to see how user behaviour and attitudes change. Web analytics are also important as forward planning can help to monitor the success of mobile library services and identify when further advertising is required. Again this is something that will be explored during the development and implementation of mobile library services within the UX2 project.

Advertisements

In November we ran a survey to all students at the university to investigate mobile demographics among students, Internet habits and attitudes towards mobile library services. The survey ran for two weeks and in that time 1,716 students responded. This was helped by an incentive prize draw of £50 Amazon vouchers – very attractive at this time of year.

In addition to demographic data and data on student’s mobile device, students were also asked about their mobile Internet habits and specifically their exposure to mobile university and library services. Finally they were asked which of the proposed mobile library services they would find most useful. The survey was loosely based on the research conducted by Information Services (IS) in March 2010 on mobile university services. Doing so allowed some comparisons to be made while also investigating mobile library services in-depth. Findings from the IS Survey can be found here: http://www.projects.ed.ac.uk/areas/itservices/integrated/ITS045/Other_documents/MobileSurvey2010.shtml.

The survey is part of the wider research within the UX2.0 project and relates to Objective 3 (deliverable 5.3) which is to evaluate the user experience in specific contexts involving real user communities. The quantitative data gathered from the survey will be supplemented by the focus group planned at the start of next year. The findings from the survey will help to shape the direction of the focus group which in turn will hopefully support the findings from the survey. Check back for a report on the focus group in the near future.

Some of the headline findings are provided below. The full report can be viewed here: http://bit.ly/ux2mobilesurvey.

  • 66.5% of students surveyed own a smartphone. This is an increase of 17.3% from the IS survey in March 2010.
  • Apple iPhones accounted for 21.9% of smart handsets, followed closely by Nokia at 20% and Samsung at 15.3%.
  • 68% of students have pay monthly contracts.
  • 74% students have either a contract that provides unlimited access to the Internet or provides sufficient access to meet their needs.
  • Services which students access online most frequently (several times a day or daily) are websites in general, email and social networks.
  • Activities which students are least likely to carry out on their mobile handsets regularly included downloading content and uploading images to photo sharing networks.
  • The biggest frustrations students experience using the Internet on mobile handsets included slow or limited Internet connection speeds, poorly designed mobile websites or websites without a mobile compatible website and the limitations of using a small screen with small buttons.
  • The highest proportion of students surveyed stated that they had not tried to access library services using their mobile device.
  • The top 3 potential University library services which students would find most useful would be:
    • Search library catalogue
    • Reserve items on loan
    • View your library record – see your charges summary and which books are reserved, requested, booked and loan

It was interesting to find that over half of students surveyed had not tried to access library services on their mobile device. It does seem that students use their mobile for other University services such as student email, the MyEd student portal and Web CT as well as other general university information such as shuttle bus timetables. Perhaps a mobile optimised library website with useful and easy accessible information would encourage students to use library services on the move more often. That coupled with effective communication that these services exist for mobile devices.

Students do seem open to the idea of mobile library services being useful, with the most useful being access to library records, the ability to search the library catalogue and databases, reserve items on loan and locate a shelf mark. However, the usefulness is dependent on the implementation of such services as many students reported that websites not optimised for smaller screens can be very difficult to use.

Limitations of smaller screen:

“It is slower than a computer and the screen is too small for a full-sized website. The library should have a special mobile website which does not take long to load and is easy to navigate on a small screen.”

Implementation of mobile websites:

“Websites should be tailored in order to be suitable for rendering on small screens. Endless scrolling makes for bad usability.”

Percentage of students who rate each library mobile service as “Very useful” or “Generally useful”

Rank % Library Service
1. 93 View your library record – see your charges summary and which books are reserved, requested, booked and loaned
2. 92.5 Search library catalogue
3. 90 Reserve items on loan
4. 89 Search library databases
5. 87 Locate a shelf mark
6. 84.5 Check PC availability in library
7. 82 Request an item through inter-library loan
8. 71 View maps of libraries
9. 67 Receive alerts relating to library information or services etc.
10. 64 Library Maps & Locations using GPS – find your way around University libraries
11. 58 Library statistics i.e. top books in different categories and popular searches
12. 55 Friend Locator – see where friends are in the library and contact them to meet up
13. 54 Read reviews others have left on items in the library
14. 52 Share items that you’ve found and/or read in the library that you think others will find useful
15. 49 Rate and review items from the library

In order to get a picture of the mobile library landscape, the mobile offering from a variety of academic institutions has been investigated. The aim of this is to understand what institutional libraries are currently doing to provide mobile services and identifying those services which The University of Edinburgh could benefit from.

North Carolina State University

NCSU have created a mobile library website which looks similar to a mobile application. The design is very similar to smart phone interfaces and this gives the website a modern look. Services which the library provide in addition to the catalogue include room booking, an instant messaging chat system with librarians and PC availability. Interestingly they also provide more unconventional services such as group finder and live webcam feeds. The group finder section helps users locate study groups in the library and the live webcams make it possible from users to monitor locations of the library remotely, including the cafe. It has been reported that the webcams generated over a third of mobile page views in the first 8 weeks that the mobile services were introduced suggesting that the webcams can be very useful to users (Meier, 2010).

Amsterdam University Library

Driek Heesakkers recently presented the case study of Amsterdam’s mobile library strategy during CILIP’s Executive briefing day, Becoming Upwardly Mobile – can libraries rise to the challenge? Amsterdam University Library‘s short-term mobile strategy resulted in a mobile web presence from a budget of €1500. This interim mobile library website provides a number of services including access to the OPAC, a simple search interface, a link to the library’s social networking page on Twitter, information on library opening hours, location, contact details and PC availability throughout the library. In addition to these basic services, the catalogue provides a simplified advanced search feature with a drop-down menu and access to the user’s library account. The interface of the search is simple with basic information in search results.

Huddersfield University and University of Bath

Huddersfield University have taken another approach to providing services on mobile devices. Using QR codes in and around their library they are able to provide links to digital information. This includes digital copies of print journals, links to mobile friendly videos and other useful digital resources. The QR codes appear at the end of book shelves, signage and on printed hand-outs. They even provide a QR code with a video on using the print credit machine (see image below). Although this technology may not be well-known among students it does have the potential to bring useful resources to students through their mobiles. As smartphones become more common place, the barrier to this technology is likely to be significantly lowered.

The University of Bath have also been using QR codes in their library. Their project explores the idea of using QR codes to help users locate items more easily by launching a map of the library on user’s mobiles. They also carried out a cross-institutional study which revealed that students are becoming more aware of QR codes and find the idea of using QR codes very appealing.

Image credit: Andrew Walsh, University of Huddersfield

Cambridge University

Cambridge University is one of the other UK university’s mentioned on M-Libraries wiki that have developed a mobile friendly site of their digital library. The mobile site focusses on allowing users to search the catalogue and access their account on the move. Notably they also highlight the benefits of logging in to the site for extra benefits including renewing items on loan, make and manage requests and build a list of items that can be emailed to any address. This would be particularly useful for research students who might otherwise write down their search results onto paper. The catalogue search is fairly simple with the addition of thumbnail images of book covers where relevant. Other useful features which are provided in the item details include a link to the Google Books preview of the full text and a map highlighting where the shelf mark is located within the library. Clearly some thought has been made to consider what information user’s need on their mobile when designing the site.

A comparitive analysis of  library mobile services in use in higher education was conducted by California Digital Library. They found that mobile library services fell into five categories 1. Library mobile website; 2. text messaging services; 3. mobile catalogue search; 4. access to resources, and 5. new tools and services. Simple information such as hours, direction, map, news and floor plans are common in many mobile library websites. Jeff Penka from OCLC supports this idea when he commented at CILIP’s Executive briefing day that “The library catalogue isn’t the only service (users) want to access via a mobile, they want to find out which computers are free, paying fines, deal with reservations, see opening hours too.”

Catalogue search is also a popular mobile service which is provided by academic digital libraries. This supports the findings from a survey conducted by Cambridge University which found that 55% of respondents were in favour of bring able to access the library catalogue from a mobile phone (Mills, 2009).

A list of resources relevant to mobile library services have been collected and compiled into a Delicious links folder. These links are open to everyone and I encourage you to add other resources I may have missed. Thanks

http://www.delicious.com/lorraine_p/bundle:UX2.0%20Mobile%20Research


Last month a usability study was carried out on the UX2 digital library prototype. The study involved 10 student participants who tried to complete a variety of tasks using the prototype. The report is now available to read in full and can be accessed via the library (http://bit.ly/ux2usabilityreport1).

The prototype is based on an open source ruby-on rails discovery interface, Blacklight which has been further developed for the project to provide additional features. Existing component services have been ‘mashed-up’ to generate the UX2.0 digital library. The prototype currently indexes the catalogues provided by the National e-Science Centre at The University of Edinburgh (NeSC) and CERN – The European Organisation for Nuclear Research. The report presents the findings of the usability testing (work package 2 – WP2.3) of the prototype which was conducted with representative users at the university. The study reveals a range of issues uncovered by observing participants using the prototype by trying to complete tasks. The findings outlined in the report provide a number of recommendations for changes to the prototype in order to improve the user’s experience.

In order to identify and fully explain the technology responsible for each issue in the report, supplementary blogs will be published on the project website in stages, as and when developmental changes are made (follow the ux2 Twitter account for announcements). It is hoped that this supplemental development documentation will make it more accessible to other digital library developers and the wider JISC community. Some of the main findings from the report are summarised below.

Positive findings from the study highlighted positive aspects of the prototype:

  • Allowing users to narrow results using the faceted navigation links was useful.
  • Providing users with details of the item content including full text preview, video and presentation slides was informative.
  • Allowing users to bookmark items and add notes was considered a useful feature.
  • Overall the layout and design was clean, simple and useful.

These positive findings from the testing are reflected in the word cloud questionnaire participants were asked to complete:

UX2 word cloud

However there were some usability issues with the prototype:

  • It was not obvious when the system saved previously selected facets in the Scope, often misleading participant’s expectations.
  • The external ‘Other’ links were not relevant to participants and often mistrusted or considered a distraction.
  • It was not clear when an item had a full text preview feature
  • Links to information resources were not prominent and often missed by participants
  • The format of text within the item details page made it difficult to read and consequently participants often ignored it.

There were also a few lessons learned from the user study which I thought would be useful to share:

  1. Recruiting participants via telephone does not necessarily guarantee attendance. Two participants did not show up to their slot after arranging the appointment with them by phone and sending them an email confirmation. However, this could also have been affected by the time of year. It transpired that many students had coursework deadlines the same week and the offending students did say they forgot because they had a heavy workload.
  2. User generated tasks are not easy to replicate using prototypes. This was not unexpected but something which was tried anyway. As suspected, it was difficult to create a task which could generate results when using such a specialised and relatively small database. However, when it was successful it did return some useful findings.
  3. It’s difficult to facilitate usability tests and log details using Morae. Any usability practitioner will tell you that it’s important concentrate on observing the participant and interacting with them and to avoid breaking the flow by stopping to take detailed notes. I found it impossible to observe a participant, engage with what they were doing and log behaviour on Morae so would recommend you recruit a note-taker if this is important for your usability study.

Screen shot of University of Edinburgh’s AquaBrowser with resource discovery services highlighted.

Screen shot of University of Edinburgh’s AquaBrowser with resource discovery services highlighted.

Background

The aim of the AquabrowserUX project was to evaluate the user experience of AquaBrowser at the University of Edinburgh (UoE). The AquaBrowser catalogue is a relatively new digital library service provided at UoE alongside the Classic catalogue  provided via Voyager which has been established at the university for a number of years. A holistic evaluation was conducted throughout with a number of activities taking place. These included a contextual enquiry of library patrons within the library environment, stakeholder interviews for persona creation and usability testing.

Intended outcome(s)

The objectives of the project were three-fold:

  1. To undertake user research and persona development. Information gathered from the contextual enquiry and stakeholder interviews were used to create a set of personas which will benefit the project and the wider JISC community. The methodologies and processes used were fully documented in the project blog.
  2. To evaluate the usefulness of resource discovery services. Contextual enquiry was conducted to engage a broader base of users. The study determined the usefulness on site and off site which will provide a more in-depth understanding of usage and behavioural patterns.
  3. To evaluate the usability of resource discovery systems. Using the personas derived from the user research, typical end users were recruited to test the usability of the AquaBrowser interface. A report was published which discusses the findings and makes recommendations on how to improve the usability of UoE’s AquaBrowser.

The challenge

There were a number of logistical issues that arose after the project kicked off. It became apparent that none of the team members had significant experience in persona development. In addition, the external commitments of subcontracted team members meant that progress was slower than anticipated. A period of learning to research established methodologies and processes for conducting interviews and analysing data took place. Consequently the persona development took longer than anticipated which delayed the recruitment of participants for usability testing (Obj3). The delay also meant that participants would be recruited during the summer months when the university is traditionally much quieter. This was a potential risk to recruit during this time but did not end up being problematic to the project. However the extension of time spent gathering qualitative data meant that it was not possible to validate the segmentation with quantitative data. This was perhaps too ambitious for a project of this scale.

The challenge of conducting the contextual enquiry within the library was to find participants willing to be observed and interviewed afterwards. The timing once again made this difficult as it took place during exams. The majority of people in the library at that time were focussed on one task which was to revise for exams. This meant that persuading them to spend ten minutes talking to researchers was understandably difficult. In addition to this, the type of users that were common in the library at that time were limited to those revising and whose needs were specific to a task and did not necessarily represent their behaviour at other times of the year.

Ensuring that real use data and participation was captured during the contextual enquiry was also a challenge. Capturing natural behaviour in context is often difficult to achieve and carries a risk of influence from the researcher. For example, to observe students in the library ethically it is necessary to inform subjects that they are being observed. However, the act of informing users may cause them to change their behaviour. In longitudinal studies the researcher is reliant on the participant self-reporting issues and behaviour, something which they are not always qualified to do effectively.

Recruitment for the persona interviews and usability testing posed a challenge not only in finding enough people but also the right type of people. Users from a range of backgrounds and differing levels of exposure to AquaBrowser who fulfil the role of one of the personas could be potentially difficult and time-consuming to fulfil. As it turned out, recruitment of staff (excluding librarians) proved to be difficult and was something that we did not manage to successfully overcome.

Established practice

Resource discovery services for libraries have evolved significantly. There is an increasing use of dynamic user interface. Faceted searching for example provides a “navigational metaphor” for boolean search operations. AquaBrowser is a leading OPAC product which provides faceted searching and new resource discovery functions in the form of their dynamic Word Cloud. Early studies have suggested a propensity of faceted searching to result in serendipitous discovery, even for domain experts.

Closer to home, The University of Edinburgh library have conducted usability research earlier this year to understand user’s information seeking behaviour and identify issues with the current digital service in order to create a more streamlined and efficient system. The National Library of Scotland has also conducted a website assessment and user research on their digital library services in 2009. This research included creating a set of personas. Beyond this, the British Library are also in the process of conducting their own user research and persona creation.

The LMS advantage

Creating a set of library personas benefits the University of Edinburgh and the wider JISC community. The characteristics and information seeking behaviour outlined in the personas have been shown to be effective templates for the successful recruitment of participants for user studies. They can also help shape future developments in library services when consulted during the design of new services. The persona hypothesis can also be carried to other institutes who may want to create their own set of personas.

The usability test report highlights a number of issues, outlined in Conclusions and Recommendations, which the university, AquaBrowser and other institutions can learn from. The methodology outlined in the report also provides guidance to those conducting usability testing for the first time and looking to embark on in-house recruitment instead of using external companies.

Key points for effective practice

  • To ensure realism of tasks in usability testing, user-generated tasks should be created with each participant.
  • Involve as many stakeholders as possible. We did not succeed in recruiting academic staff and were therefore unable to evaluate this user group however, the cooperation with Information Services through project member Liza Zamboglou did generate positive collaboration with the library during the contextual enquiry, persona interviews and usability testing.
  • Findings from the user research and usability testing suggest that resource discovery services provided by AquaBrowser for UoE can be improved in order to be useful and easy to operate.
  • Looking back over the project and the methods used to collect user data we found that contextual enquiry is a very useful method of collecting natural user behaviour when used in conjunction with other techniques such as interviews and usability tests.
  • The recruitment of participants was successful despite the risks highlighted above. The large number of respondents demonstrated that recruitment of students is not difficult when a small incentive is provided and can be achieved at a much lower cost than if a professional recruitment company had been used.
  • It is important to consider the timing of any recruitment before undertaking a user study. To maximise potential respondents, it is better to recruit during term time than between terms or during quieter periods. Although the response rate during the summer was still sufficient for persona interviews, the response rate during the autumn term was much greater. Academic staff should also be recruited separately through different streams in order to ensure all user groups are represented.

Conclusions and recommendations

Overall the project outcomes from each of the objectives have been successfully delivered. The user research provided a great deal of data which enabled a set of personas to be created. This artifact will be useful to UoE digital library by providing a better understanding of its users. This will come in handy when embarking on any new service design. The process undertaken to create the personas was also fully documented and this in itself is a useful template for others to follow for their own persona creation.

The usability testing has provided a report (Project Posts and Resources) which clearly identifies areas where the AquaBrowser catalogue can be improved. The usability report makes recommendations that if implemented has potential to improve the user experience of UoE AquaBrowser. Based on the findings from the usability testing and contextual enquiry, it is clear that the contextual issue and its position against the other OPAC (Classic) must be resolved. The opportunity for UoE to add additional services such as an advanced search and bookmarking system would also go far in improving the experience. We recommend that AquaBrowser and other institutes also take a look at the report to see where improvements can be made. Evidence from the research found that the current representation of the Word Cloud is a big issue and should be addressed.

The personas can be quantified and used against future recruitment and design. All too often users are considered too late in a design (or redesign and restructuring) process. Assumptions are made about ‘typical’ users which are based more opinion than in fact. With concrete research behind comprehensive personas it is much easier to ensure that developments will benefit the primary user group.

Additional information

Project Team

  • Boon Low, Project Manager, Developer, boon.low@ed.ac.uk – University of Edinburgh National e-Science Centre
  • Lorraine Paterson, Usability Analyst, l.paterson@nesc.ac.uk – University of Edinburgh National e-Science Centre
  • Liza Zamboglou, Usability Consultant, liza.zamboglou@ed.ac.uk – Senior Manager , University of Edinburgh Information Services
  • David Hamill, Usability Specialist, web@goodusability.co.uk – Self-employed

Project Website

PIMS entry

Project Posts and Resources

Project Plan

Conference Dissemination

User Research and Persona Development (Obj1)

Usefulness of Resource Discovery Services (Obj2)

Usability of Resource Discover Services (Obj3)

AquabrowserUX Final Project Post

Last week during our regular project meetings we decided on a roadmap for the user research aspect for the last few months of the project (is it that time already?!). In the spirit of transparency I decided to publish my plan here. I’ll of course be blogging along the way, reporting findings and evaluating the work as I go. If you can point me to any existing research that I should include in my investigation, please feel free to leave a comment with a link at the end and thanks in advance!

Background

This research will examine the current use of Web 2.0 developments in digital libraries, their use scenarios and applications. The scope of the remaining research will be centred on services which diffuse the digital library through mobile technology. The aim of this mobile research is to identify areas that may be relevant to The University of Edinburgh (UoE) and evaluate a prototype mobile library service.

Project Aim

To enhance the user experience of digital libraries through technological developments centred on usability inspection and evaluation.

Project Objectives

  1. To undertake usability inspection and contemporary UX techniques research
  2. To enhance digital libraries with state-of-the-art technologies
  3. To evaluate user experience in specific contexts involving real user communities

Mobile Research Aims:

1. Review current mobile digital library landscape, how services are diffused using mobile platforms and what UoE can learn (Obj3)

Formative evaluation informed by existing mobile digital library services and usability studies of mobile library services will be undertaken. This will help to provide a clear picture of the mobile digital library landscape which will inform the project’s own development work. Existing mobile usability research[i] will also provide insights into existing user centred design processes which can be adapted for this project.

Output: Blog series reviewing the trends in mobile digital library services, highlighting successful services and identifying what UoE can learn from other projects. In addition, a list of existing mobile digital library resources will be created as a resource for others.

2. Review good UX practices for mobile applications & websites as well as usability evaluation techniques (Obj1)

As mobile usability is a relatively new subject to the project, research will be conducted on usability practices for mobile design and development.  In addition, mobile evaluation methodologies will be identified and incorporated into the prototype evaluation (Obj2).

Output: Blog post which highlights good mobile UX resources and describes the evaluation technique which will be applied to the project.

3. Investigate what users want from a mobile library service (Obj3)

Continuing on from the Mobile Services survey conducted by Information Services (IS) in March 2010[ii], a subsequent survey will be conducted with UoE students which will focus on mobile library services. The findings will provide insight into the types of services users would find useful and this will hopefully influence the direction of development. The research will also help to support the ongoing mobile services development by IS as it will provide additional data which can be benchmarked against their previous survey.

The quantitative data gathered from the survey will be supplemented with a focus group with those likely to be using the service (end users) and those helping to provide the service (staff, developers, librarians). The findings will not only help to qualify the findings from the survey but also provide a broad perspective of how a digital library service should be shaped by including all stakeholders.

Output: Survey report detailing findings and outcomes from first focus group with stakeholders.

4. Evaluate the usability of the prototype mobile library service (Obj1)

The usability of the higher fidelity prototype will be evaluated with representative users. These one-on-one sessions will take place with a small number of users (6-12) and will be conducted using a simulator, a smart phone or the user’s own mobile handset. Qualitative date will be captured and reported. The objective of this usability study is to ensure the success of the prototype and provide a use case for digital library services at the University of Edinburgh and beyond.

Output: Summary of findings from focus group and detailed usability test report.


When carrying out usability studies on search interfaces, it’s often better to favour interview-based tasks over pre-defined ‘scavenger-hunt’ tasks. In this post I’ll explain why this is the case and why you may have to sacrifice capturing metrics in order to achieve this realism.

In 2006, Jared Spool of User Interface Engineering wrote an article entitled Interview-Based Tasks: Learning from Leonardo DiCaprio in it he explains that it often isn’t enough to create test tasks that ask participants to find a specific item on a website. He calls such a task a Scavenger-Hunt task. Instead he introduces the idea of interview-based tasks.

When testing the search interface for a library catalogue, a Scavenger Hunt task might read:

You are studying Russian Literature and your will be reading Leo Tolstoy soon. Find the English version of Tolstoy’s ‘War and Peace’ in the library catalogue.

I’ll refer to this as the Tolstoy Task in this post. Most of your participants (if they’re university students) should have no trouble understanding the task. But it probably won’t feel real to any of them. Most of them will simply type ‘war and peace’ into the search and see what happens.

Red routes

The Tolstoy Task is not useless, you’ll probably still witness things of interest. So it’s better than having no testing at all.

But it answers only one question – When users know the title of the book, author and how to spell them both correctly, how easy is it to find the English version of Leo Tolstoy’s War and Peace?

A very specific question like this can still be useful for many websites. For example a car insurance company could ask – When the user has all of his vehicle documents in front of him, how easy is it for them to get a quote from our website?

Answering this question would give them a pretty good idea of how well their website was working. This is because it’s probably the most important journey on the site. Most websites have what Dr David Travis calls Red Routes – the key journeys on a website. When you measure the usability of a website’s red routes you effectively measure the usability of the site.

However many search interfaces such as that for a university library catalogue, don’t have one or two specific tasks that are more important than any others. It’s possible to categorise tasks but difficult to introduce them into a usability test without sacrificing a lot of realism.

Interview-based tasks

The interview-based task is Spool’s answer to the shortfalls of the Scavenger Hunt task. This is where you create a task with the input of the participant and agree what successful completion of the task will mean before they begin.

When using search interfaces, people often develop search tactics based upon the results they are being shown. As a result they can change tactics several times. They can change their view of the problem based upon the feedback they are getting.

Whilst testing the Aquabrowser catalogue for the University of Edinburgh, participants helped me to create tasks that I’d never have been able to do so on my own. Had we not done this, I wouldn’t have been able to observe their true behaviour.

One participant used the search interface to decide her approach to an essay question. Together we created a task scenario where she was given an essay to write on National identity in the work of Robert Louis Stevenson.

She had decided that the architecture in Jekyll and Hyde whilst set in London, had reminded her more of Edinburgh. She searched for sources that referred to Edinburgh’s architecture in Scottish literature, opinion on architecture in Stevenson’s work and opinion on architecture in national identity.

The level of engagement she had in the task allowed me to observe behaviour that a pre-written task would never have been able to do.

It also made no assumptions about how she uses the interface. In the Tolstoy task, I’d be assuming that people arrive at the interface with a set amount of knowledge. In an interview-based task I can establish how much knowledge they would have about a specific task before they use the interface. I simply ask them.

Realism versus measurement

The downside to using such personalised tasks is that it’s very difficult to report useful measurements. When you pre-define tasks you know that each participant will perform the same task. So you can measure the performance of that task. By doing this you can ask “How usable is this interface?” and provide an answer.

With interview-based tasks this is often impossible because the tasks vary in subject and complexity. It’s often  then inappropriate to use them to provide an overall measure of usability.

Exposing issues

I believe that usability testing is more reliable as a method for exposing issues than it is at providing a measure of usability. This is why I favour using interview-based tasks in most cases.

It’s difficult to say how true to life the experience you’re watching is. If they were sitting at home attempting a task then there’d be nobody watching them and taking notes. Nobody would be asking them to think aloud and showing interest in what they were doing. So if they fail a task in a lab, can you be sure they’d fail it at home?

But for observing issues I feel it’s more reliable. If participants misunderstand something about the interface in a test, you can be fairly sure that someone at home will be making that same misunderstanding.

And it can never hurt to make something more obvious.

del.icio.us bookmarks

Twitter feed

Archive