Researching Usability

Posts Tagged ‘report’

Mobile library survey report

Posted by: on: December 15, 2010

In November we ran a survey to all students at the university to investigate mobile demographics among students, Internet habits and attitudes towards mobile library services. The survey ran for two weeks and in that time 1,716 students responded. This was helped by an incentive prize draw of £50 Amazon vouchers – very attractive at this time of year.

In addition to demographic data and data on student’s mobile device, students were also asked about their mobile Internet habits and specifically their exposure to mobile university and library services. Finally they were asked which of the proposed mobile library services they would find most useful. The survey was loosely based on the research conducted by Information Services (IS) in March 2010 on mobile university services. Doing so allowed some comparisons to be made while also investigating mobile library services in-depth. Findings from the IS Survey can be found here: http://www.projects.ed.ac.uk/areas/itservices/integrated/ITS045/Other_documents/MobileSurvey2010.shtml.

The survey is part of the wider research within the UX2.0 project and relates to Objective 3 (deliverable 5.3) which is to evaluate the user experience in specific contexts involving real user communities. The quantitative data gathered from the survey will be supplemented by the focus group planned at the start of next year. The findings from the survey will help to shape the direction of the focus group which in turn will hopefully support the findings from the survey. Check back for a report on the focus group in the near future.

Some of the headline findings are provided below. The full report can be viewed here: http://bit.ly/ux2mobilesurvey.

  • 66.5% of students surveyed own a smartphone. This is an increase of 17.3% from the IS survey in March 2010.
  • Apple iPhones accounted for 21.9% of smart handsets, followed closely by Nokia at 20% and Samsung at 15.3%.
  • 68% of students have pay monthly contracts.
  • 74% students have either a contract that provides unlimited access to the Internet or provides sufficient access to meet their needs.
  • Services which students access online most frequently (several times a day or daily) are websites in general, email and social networks.
  • Activities which students are least likely to carry out on their mobile handsets regularly included downloading content and uploading images to photo sharing networks.
  • The biggest frustrations students experience using the Internet on mobile handsets included slow or limited Internet connection speeds, poorly designed mobile websites or websites without a mobile compatible website and the limitations of using a small screen with small buttons.
  • The highest proportion of students surveyed stated that they had not tried to access library services using their mobile device.
  • The top 3 potential University library services which students would find most useful would be:
    • Search library catalogue
    • Reserve items on loan
    • View your library record – see your charges summary and which books are reserved, requested, booked and loan

It was interesting to find that over half of students surveyed had not tried to access library services on their mobile device. It does seem that students use their mobile for other University services such as student email, the MyEd student portal and Web CT as well as other general university information such as shuttle bus timetables. Perhaps a mobile optimised library website with useful and easy accessible information would encourage students to use library services on the move more often. That coupled with effective communication that these services exist for mobile devices.

Students do seem open to the idea of mobile library services being useful, with the most useful being access to library records, the ability to search the library catalogue and databases, reserve items on loan and locate a shelf mark. However, the usefulness is dependent on the implementation of such services as many students reported that websites not optimised for smaller screens can be very difficult to use.

Limitations of smaller screen:

“It is slower than a computer and the screen is too small for a full-sized website. The library should have a special mobile website which does not take long to load and is easy to navigate on a small screen.”

Implementation of mobile websites:

“Websites should be tailored in order to be suitable for rendering on small screens. Endless scrolling makes for bad usability.”

Percentage of students who rate each library mobile service as “Very useful” or “Generally useful”

Rank % Library Service
1. 93 View your library record – see your charges summary and which books are reserved, requested, booked and loaned
2. 92.5 Search library catalogue
3. 90 Reserve items on loan
4. 89 Search library databases
5. 87 Locate a shelf mark
6. 84.5 Check PC availability in library
7. 82 Request an item through inter-library loan
8. 71 View maps of libraries
9. 67 Receive alerts relating to library information or services etc.
10. 64 Library Maps & Locations using GPS – find your way around University libraries
11. 58 Library statistics i.e. top books in different categories and popular searches
12. 55 Friend Locator – see where friends are in the library and contact them to meet up
13. 54 Read reviews others have left on items in the library
14. 52 Share items that you’ve found and/or read in the library that you think others will find useful
15. 49 Rate and review items from the library
Tags: , , , , ,

UX2.0 Usability Test Report

Posted by: on: November 12, 2010

Last month a usability study was carried out on the UX2 digital library prototype. The study involved 10 student participants who tried to complete a variety of tasks using the prototype. The report is now available to read in full and can be accessed via the library (http://bit.ly/ux2usabilityreport1).

The prototype is based on an open source ruby-on rails discovery interface, Blacklight which has been further developed for the project to provide additional features. Existing component services have been ‘mashed-up’ to generate the UX2.0 digital library. The prototype currently indexes the catalogues provided by the National e-Science Centre at The University of Edinburgh (NeSC) and CERN – The European Organisation for Nuclear Research. The report presents the findings of the usability testing (work package 2 – WP2.3) of the prototype which was conducted with representative users at the university. The study reveals a range of issues uncovered by observing participants using the prototype by trying to complete tasks. The findings outlined in the report provide a number of recommendations for changes to the prototype in order to improve the user’s experience.

In order to identify and fully explain the technology responsible for each issue in the report, supplementary blogs will be published on the project website in stages, as and when developmental changes are made (follow the ux2 Twitter account for announcements). It is hoped that this supplemental development documentation will make it more accessible to other digital library developers and the wider JISC community. Some of the main findings from the report are summarised below.

Positive findings from the study highlighted positive aspects of the prototype:

  • Allowing users to narrow results using the faceted navigation links was useful.
  • Providing users with details of the item content including full text preview, video and presentation slides was informative.
  • Allowing users to bookmark items and add notes was considered a useful feature.
  • Overall the layout and design was clean, simple and useful.

These positive findings from the testing are reflected in the word cloud questionnaire participants were asked to complete:

UX2 word cloud

However there were some usability issues with the prototype:

  • It was not obvious when the system saved previously selected facets in the Scope, often misleading participant’s expectations.
  • The external ‘Other’ links were not relevant to participants and often mistrusted or considered a distraction.
  • It was not clear when an item had a full text preview feature
  • Links to information resources were not prominent and often missed by participants
  • The format of text within the item details page made it difficult to read and consequently participants often ignored it.

There were also a few lessons learned from the user study which I thought would be useful to share:

  1. Recruiting participants via telephone does not necessarily guarantee attendance. Two participants did not show up to their slot after arranging the appointment with them by phone and sending them an email confirmation. However, this could also have been affected by the time of year. It transpired that many students had coursework deadlines the same week and the offending students did say they forgot because they had a heavy workload.
  2. User generated tasks are not easy to replicate using prototypes. This was not unexpected but something which was tried anyway. As suspected, it was difficult to create a task which could generate results when using such a specialised and relatively small database. However, when it was successful it did return some useful findings.
  3. It’s difficult to facilitate usability tests and log details using Morae. Any usability practitioner will tell you that it’s important concentrate on observing the participant and interacting with them and to avoid breaking the flow by stopping to take detailed notes. I found it impossible to observe a participant, engage with what they were doing and log behaviour on Morae so would recommend you recruit a note-taker if this is important for your usability study.
Tags: ,

AquabrowserUX Final Project Post

Posted by: on: October 20, 2010

Screen shot of University of Edinburgh’s AquaBrowser with resource discovery services highlighted.

Screen shot of University of Edinburgh’s AquaBrowser with resource discovery services highlighted.

Background

The aim of the AquabrowserUX project was to evaluate the user experience of AquaBrowser at the University of Edinburgh (UoE). The AquaBrowser catalogue is a relatively new digital library service provided at UoE alongside the Classic catalogue  provided via Voyager which has been established at the university for a number of years. A holistic evaluation was conducted throughout with a number of activities taking place. These included a contextual enquiry of library patrons within the library environment, stakeholder interviews for persona creation and usability testing.

Intended outcome(s)

The objectives of the project were three-fold:

  1. To undertake user research and persona development. Information gathered from the contextual enquiry and stakeholder interviews were used to create a set of personas which will benefit the project and the wider JISC community. The methodologies and processes used were fully documented in the project blog.
  2. To evaluate the usefulness of resource discovery services. Contextual enquiry was conducted to engage a broader base of users. The study determined the usefulness on site and off site which will provide a more in-depth understanding of usage and behavioural patterns.
  3. To evaluate the usability of resource discovery systems. Using the personas derived from the user research, typical end users were recruited to test the usability of the AquaBrowser interface. A report was published which discusses the findings and makes recommendations on how to improve the usability of UoE’s AquaBrowser.

The challenge

There were a number of logistical issues that arose after the project kicked off. It became apparent that none of the team members had significant experience in persona development. In addition, the external commitments of subcontracted team members meant that progress was slower than anticipated. A period of learning to research established methodologies and processes for conducting interviews and analysing data took place. Consequently the persona development took longer than anticipated which delayed the recruitment of participants for usability testing (Obj3). The delay also meant that participants would be recruited during the summer months when the university is traditionally much quieter. This was a potential risk to recruit during this time but did not end up being problematic to the project. However the extension of time spent gathering qualitative data meant that it was not possible to validate the segmentation with quantitative data. This was perhaps too ambitious for a project of this scale.

The challenge of conducting the contextual enquiry within the library was to find participants willing to be observed and interviewed afterwards. The timing once again made this difficult as it took place during exams. The majority of people in the library at that time were focussed on one task which was to revise for exams. This meant that persuading them to spend ten minutes talking to researchers was understandably difficult. In addition to this, the type of users that were common in the library at that time were limited to those revising and whose needs were specific to a task and did not necessarily represent their behaviour at other times of the year.

Ensuring that real use data and participation was captured during the contextual enquiry was also a challenge. Capturing natural behaviour in context is often difficult to achieve and carries a risk of influence from the researcher. For example, to observe students in the library ethically it is necessary to inform subjects that they are being observed. However, the act of informing users may cause them to change their behaviour. In longitudinal studies the researcher is reliant on the participant self-reporting issues and behaviour, something which they are not always qualified to do effectively.

Recruitment for the persona interviews and usability testing posed a challenge not only in finding enough people but also the right type of people. Users from a range of backgrounds and differing levels of exposure to AquaBrowser who fulfil the role of one of the personas could be potentially difficult and time-consuming to fulfil. As it turned out, recruitment of staff (excluding librarians) proved to be difficult and was something that we did not manage to successfully overcome.

Established practice

Resource discovery services for libraries have evolved significantly. There is an increasing use of dynamic user interface. Faceted searching for example provides a “navigational metaphor” for boolean search operations. AquaBrowser is a leading OPAC product which provides faceted searching and new resource discovery functions in the form of their dynamic Word Cloud. Early studies have suggested a propensity of faceted searching to result in serendipitous discovery, even for domain experts.

Closer to home, The University of Edinburgh library have conducted usability research earlier this year to understand user’s information seeking behaviour and identify issues with the current digital service in order to create a more streamlined and efficient system. The National Library of Scotland has also conducted a website assessment and user research on their digital library services in 2009. This research included creating a set of personas. Beyond this, the British Library are also in the process of conducting their own user research and persona creation.

The LMS advantage

Creating a set of library personas benefits the University of Edinburgh and the wider JISC community. The characteristics and information seeking behaviour outlined in the personas have been shown to be effective templates for the successful recruitment of participants for user studies. They can also help shape future developments in library services when consulted during the design of new services. The persona hypothesis can also be carried to other institutes who may want to create their own set of personas.

The usability test report highlights a number of issues, outlined in Conclusions and Recommendations, which the university, AquaBrowser and other institutions can learn from. The methodology outlined in the report also provides guidance to those conducting usability testing for the first time and looking to embark on in-house recruitment instead of using external companies.

Key points for effective practice

  • To ensure realism of tasks in usability testing, user-generated tasks should be created with each participant.
  • Involve as many stakeholders as possible. We did not succeed in recruiting academic staff and were therefore unable to evaluate this user group however, the cooperation with Information Services through project member Liza Zamboglou did generate positive collaboration with the library during the contextual enquiry, persona interviews and usability testing.
  • Findings from the user research and usability testing suggest that resource discovery services provided by AquaBrowser for UoE can be improved in order to be useful and easy to operate.
  • Looking back over the project and the methods used to collect user data we found that contextual enquiry is a very useful method of collecting natural user behaviour when used in conjunction with other techniques such as interviews and usability tests.
  • The recruitment of participants was successful despite the risks highlighted above. The large number of respondents demonstrated that recruitment of students is not difficult when a small incentive is provided and can be achieved at a much lower cost than if a professional recruitment company had been used.
  • It is important to consider the timing of any recruitment before undertaking a user study. To maximise potential respondents, it is better to recruit during term time than between terms or during quieter periods. Although the response rate during the summer was still sufficient for persona interviews, the response rate during the autumn term was much greater. Academic staff should also be recruited separately through different streams in order to ensure all user groups are represented.

Conclusions and recommendations

Overall the project outcomes from each of the objectives have been successfully delivered. The user research provided a great deal of data which enabled a set of personas to be created. This artifact will be useful to UoE digital library by providing a better understanding of its users. This will come in handy when embarking on any new service design. The process undertaken to create the personas was also fully documented and this in itself is a useful template for others to follow for their own persona creation.

The usability testing has provided a report (Project Posts and Resources) which clearly identifies areas where the AquaBrowser catalogue can be improved. The usability report makes recommendations that if implemented has potential to improve the user experience of UoE AquaBrowser. Based on the findings from the usability testing and contextual enquiry, it is clear that the contextual issue and its position against the other OPAC (Classic) must be resolved. The opportunity for UoE to add additional services such as an advanced search and bookmarking system would also go far in improving the experience. We recommend that AquaBrowser and other institutes also take a look at the report to see where improvements can be made. Evidence from the research found that the current representation of the Word Cloud is a big issue and should be addressed.

The personas can be quantified and used against future recruitment and design. All too often users are considered too late in a design (or redesign and restructuring) process. Assumptions are made about ‘typical’ users which are based more opinion than in fact. With concrete research behind comprehensive personas it is much easier to ensure that developments will benefit the primary user group.

Additional information

Project Team

  • Boon Low, Project Manager, Developer, boon.low@ed.ac.uk – University of Edinburgh National e-Science Centre
  • Lorraine Paterson, Usability Analyst, l.paterson@nesc.ac.uk – University of Edinburgh National e-Science Centre
  • Liza Zamboglou, Usability Consultant, liza.zamboglou@ed.ac.uk – Senior Manager , University of Edinburgh Information Services
  • David Hamill, Usability Specialist, web@goodusability.co.uk – Self-employed

Project Website

PIMS entry

Project Posts and Resources

Project Plan

Conference Dissemination

User Research and Persona Development (Obj1)

Usefulness of Resource Discovery Services (Obj2)

Usability of Resource Discover Services (Obj3)

AquabrowserUX Final Project Post

Tags: , , , , , ,

del.icio.us bookmarks

Archive

Blog at WordPress.com.