Researching Usability

Posts Tagged ‘prototype testing


Following on from the usability testing of the desktop prototype digital library, we conducted more user research on the mobile prototype. The prototype is similar to the full version but with some services removed to create a simpler interface. The facet navigation situated in the right column is now provided within a ‘Refine’ link. The ability to bookmark items is also available. As before, the prototype is based on an open source ruby-on rails discovery interface, Blacklight which has been further developed throughout the project. The prototype indexes the catalogues provided by the National e-Science Centre at The University of Edinburgh (NeSC) and CERN – The European Organisation for Nuclear Research.

The data capture and test methods are detailed here alongside the main findings.


On the 10th and 11th March 2011, usability testing was conducted on the UX2 mobile optimised digital library prototype with six selected students from the University of Edinburgh (UoE).  Each test was carried out as a one-to-one session and comprised an explanation of the research, undertaking task-based scenarios followed by a short post-test interview and word-choice questionnaire. A full description of the prototype and the changes made to the desktop version to optimise it for mobile devices will be documented shortly on the associate project blog, Enhancing User Interactions in Digital Libraries.

View of mobile device from webcam setup

Six participants were recruited from the same list which was originally compiled for the focus group recruitment in January. Only those who owned an iPhone were invited to take part as this was the platform the prototype had been optimised for. Each participant was given a £10 Amazon voucher as payment for their time. Each session lasted between 50 and 65 minutes. A few additional statistics on the participant’s profiles are provided below:

  • All 6 participants owned an Apple device (iPhone 3GS, iPhone 4 or iPod touch)
  • 3 participants had attended our focus group in January, 3 had not
  • 4 participants were undergraduates, 2 were postgraduates
  • 50:50 male to female ratio
  • All were on a pay monthly contract for their mobile phone device

Complete setup with webcam positioned on perspex above phone. Also the additional webcam to capture body language.

To record the testing we used a similar set up to that suggested by Harry Brignull on his blog. It was fairly low-cost (approx £80 excluding Morae software) and required two webcams, a small piece of perspex (approx 35cm x 10cm), testing software (we used Morae v3 as it allowed us to capture from 2 webcams), cheap plastic mobile phone cases for each type of handset being tested (two in this case) and velcro to attach them to the perspex. It was very easy to mould the perspex to shape by heating it gently above something readily available, a toaster! The Logitech C905 webcam we used had a great advanced settings which allowed us to mirror and flip the image, making it easy to decipher what was going on (see image). Overall the setup worked well as it was lightweight and relatively unobtrusive. The camera was positioned in the best place at all times and this allowed us to see exactly what was going on as well as record participant’s interactions and body language using a second webcam.

Task scenarios

The scenarios used were adapted from those used for the desktop usability tests. They were designed to test a variety of the prototype’s features. The main focus was the facet navigation system (Refine link), the presentation of information in the item detail page and the bookmarking service:

  1. As part of your coursework your lecturer has asked you to read a recent presentation on fusion. Using the prototype can you find a suitable presentation published in the last 2-3 years.
  2. You have to write an essay on quantum mechanics, can you use the prototype to find several resources which will help you get started? If you wanted to save these items to refer to later when you’re at a computer, how would you do this?
  3. You are looking for a paper on Grid Computing to read on the bus. Can you use the prototype to find a suitable digital resource?
  4. You are looking for a presentation given by Harald Bath at the EGEE summer school; can you find the video or audio presentation to watch at the gym?

Word Choice Results

The aim of the word choice exercise was to elicit an overall impression of the participant’s experience using the prototype. The word choice exercise presented participants with a list of 97 adjectives (both positive and negative) at the end of the session and asked them to pick those they felt were descriptive of the prototype

Word cloud results

Each of the words shown in image above represents a word that was ticked at least once. The larger and darker the word is, the more often it was ticked by participants.

The most prominent words were all positive: Accessible, Straightforward, Easy to use, Clean and Useful. These words were chosen by 5 of the 6 participants surveyed. Other positive words include Clear, Convenient, Effective, Efficient, Flexible, Relevant and Usable. Some of these words are a particularly interesting as they could directly relate to the ability to access the service anywhere using a mobile device.

Some of the negative words selected include: Simplistic (could also be positive), Ordinary, Ambiguous, Inadequate, Ineffective, Old and Confusing. However some of these words were used by a minority of participants and were therefore not as visible. The words are not all that surprising considering the mobile website is a prototype with fewer services than the desktop version.


Finding Information (Navigation and refine)

Some students noticed that there was no universal ‘Home’ link throughout the site. Students felt that a shortcut back to the home page would be a good idea, despite the fact that the search field was present on the results page. This suggests a desire from students to be able to skip pages instead of navigating back through pages one at a time.

NCSU Mobile search form

Many of the students stated a need to have an advanced search facility. Often students were looking for either an Advanced search link near the search field or a drop-down menu where they could specify details such as Author, Title, Subject/Keyword. As this is a feature they encounter in other search services, they still expect to have access to it on a mobile device. An example of a mobile library site providing such a service is North Carolina State University (see image).

The desire for an advanced search was  problematic for users who did not see the ‘Refine’ link at the top of the page. Those users struggled to complete tasks and consequently their experience of the prototype was affected. Failing to use the Refine service made using the prototype much more difficult. This was demonstrated when students were asked to search for items from a specific year (task 1). Students had no way of knowing if they had searched through every relevant result without looking at every page of results. This finding suggests that it is imperative that a solution to the Refine service be provided. When the Refine link was pointed out to students they stated that they did not see the link due to its location at the top of the screen or did not fully understand its label. These students were searching for an ‘Advanced’ label located close to the search form and search results.

Those students who did use the refine service wanted to filter by more than one item at a time. This was particularly evident in task 1 when students were searching for presentations published in the last 2-3 years. Students found it quite laborious to select one year at a time to review the results. They wanted to be able to search a period of time, either the last 5 years or set the search terms themselves (possibly using a form or sliders). This finding was also revealed during the desktop prototype testing. It demonstrates that this is an important criteria for students to make searching easier and remains so when using the service on a mobile device.

The refine service itself was generally well received when used. The facets provided were considered useful to students and listed in a logical order of importance. One student realised that the items listed in the year facet was not necessarily the last ten years, rather the top ten results. This caused some confusion as first and could affect the success of student’s tasks. In addition, students found using the refine service laborious at times with unnecessary steps involved which could prove troublesome on a mobile device. An idea of the task flow is provided below to outline the steps involved in a typical task using the refine service:

Select ‘Refine’ > Select ‘2010’ from Year facet > [view results] > Go back to ‘Refine’ > Remove 2010 > [view results] > Go back to ‘Refine’ > Select ‘2009’…


Select ‘Refine’ > Select ‘2010’ >[view results] > Remove 2010 from results page > [view results] > Go back to ‘Refine’ > Select ‘2009’…

Note: actions in square brackets happen automatically in the process.

Several students used the first system to remove facets perhaps because they did not notice the option to do this on the results page. Consequently, the additional number of steps involved appeared to affect the user’s experience. This was also one of the reasons that students wanted to be able to select a range of dates at once instead of going backwards and forwards within the refine service. One student suggested a tick box option where they could select all the items within each facet they wanted to use to narrow their results at once.

Saving Information (Bookmarking service)

When students were asked to save information to refer to later as part of task 2, some used the bookmarking service. Those who did found it relatively easy to bookmark an item and retrieve it afterwards. The feedback seemed to suggest that it was clear when an item had been saved and the folder icon at the top of the screen was clearly visible. However, there seems to be a bug when a user tries to bookmark an item before logging in. Upon selecting the ‘Bookmark this’ link and completing the login form, users are taken to their bookmarks page without the item listed. The user has to go back to the results page and attempt the action again for it to be successful. In addition, although users could see the number of items bookmarked next to the folder icon, it might be more difficult to spot under different lighting conditions. The white text on pale green background makes less likely to stand out.

There was also a desire for additional features within the bookmarking service which would make it a more effective tool. Additional information on each item including author, date, location and most importantly, shelfmark, would make it easier for users to distinguish items with the same title and locate a number of saved items in the library quickly. Students also wanted to be able to categorise bookmarks into sections, similar to the folder system in a browser’s bookmarking service. Being able to export bookmarks by emailing them to an address was also considered a useful feature to provide.

Although the usability of the bookmarking service was considered to be good, the usefulness was not apparent to every student. Many students have their own system in place for recording items of interest in the library. Writing details down on paper, copy & pasting text to a separate document or simply minimising the browser to open again at the relevant moment. The Safari browser on iPhones allow users to have several windows open at once and automatically saves the state of all windows when it is closed. One participant in particular used this system to park information in order to retrieve it when required. Consequently they did not envisage themself using the bookmarking system. It is interesting to note that of those students who participated in the study, none of them stated that they used the existing bookmarking system available in the University’s own library website called ‘My List (Bookbag)’ .

Reviewing Information

Although the UX2 library is a prototype, students felt that the level of information provided on each item was for the most part adequate. Being able to view documents was often expected by students but was not always possible. One student questioned whether everyone would be able to preview documents if they did not have a Google account as this is requested upon selecting the electronic document. This could be problematic, especially because feedback from the focus groups indicated averseness to downloading files to mobile devices due to limited data storage.

In addition to previewing documents, links to information sources under the heading ‘Links, Files’ was not easily understood by students. The links were not easy to identify because they are not presented as conventional blue, underlined text. The long URLs and small text size also made it difficult for students to guess where the links might lead. In many situations, the links did not meet user expectations. Students would select a link expecting it to lead them to the full text item when instead it went to an external website. Students wanted to be warned when links lead to an external (and often not optimised for mobile) website. Although this was an issue during the desktop usability tests, it became even more apparent among students using the prototype on a smaller screen.

Something which was requested by students not only during the usability tests but also in the focus groups was the ability to easily access a map of libraries. Having such information would make it much easier to locate books, particularly when students are not familiar with the particular library. Students felt a link to such a map could be provided on the item information page and located under the library holding data to make it easy to access. There was also a desire to provide a simple system which informs students when an item is available or on loan. A colour coding system or simple icon was suggested which could be displayed in the results page next to each item – green for available, red for on loan. A library which has gone some way to address this need is NCSU. They provide users with the opportunity to filter out items which are not available (see earlier image).

Post test interview

Overall students felt that the prototype was fairly effective in helping them find information. Those who did not see the ‘Refine’ link naturally believed it could have been clearer. Another student stated that quality of results was sometimes an issue, suggesting the need for improvement. The timeliness of resources was often dependent on the subject students were studying. Subjects where research tends to move quickly, such as technology, recent publications are much more important. Students were asked to state which two things which they particularly liked about the prototype. Their answers are listed below:

  • Refine page (2)
  • Item information page
  • Level of information provided – not overloaded
  • Font style is modern
  • Minimalist and contemporary design (2)
  • Simple search
  • Bookmarking system (2)
  • Being able to filter results by type e.g. book, presentation

Some things that students believed could be improved:

  • Refine search – visibility and task flow
  • Design, particularly the home page – no logo, name or clear description of purpose (2)
  • Provide a link to home page throughout the site (2)
  • Provide search options next to search form
  • Improve the date range of the Date facet
  • Tips to guide you through the website
  • Visibility and source of links on item page
  • Provide an additional option to search/narrow results by library


Observation of the usability tests showed that participants coped well undertaking tasks using a smaller screen. The biggest issue was the visibility of the refine page which contained the facet navigation service. When participants were not aware of this option, their experience of using the prototype was severely compromised. Those who did use the refine service were able to complete tasks more efficiently but did find the number of steps involved to do so unnecessary. This suggests further work is required on the implementation of a facet navigation service to improve its usefulness and usability. Although some of the students appreciated the minimalist nature of the prototype, there was still desire to undertake more than just simple searches. The bookmarking service was on the whole well received and was considered useful with the addition of a few more features. However, the uptake of such a service is still unknown as students often had existing bookmarking systems in place.

As stated in the previous blog post, the second phase of guerilla testing was conducted on 30th June. It was hoped that in the second phase we would have the opportunity to test with more people as the course finished before lunch. This did appear to be the case as I managed to test 4 individuals who were happy to stay up to 30 minutes in some cases to take part.

Before the testing took place some design changes were made to the prototype based on the feedback from the first phase. Based on the findings a number of recommended changes were made (see Recommendations on project Wiki page). Within the time scale it was possible to implement the following changes before phase two testing:

  • Ensure that categories within the Year facet are presented in a chronological order, most recent first.
  • Provide an additional facet to include Author.
  • Allow users to de-select facets within the facet navigation and not just using the breadcrumb system at the top.

The test plan remained the same and again I alternated the websites between each participant to reduce bias. The most startling finding in phase 2 was the user’s preferred site. Three of the four participants preferred the prototype in one way or another to the NeSC Library. With such small numbers it’s difficult to say that this is a trend, but it is an improvement on the first round of testing. It would be interesting to see if this pattern continues during Phase 3 in August and indeed with the focus group planned in September.

There was more evidence to support a faceted navigation with expanded facet values. Some of the users commented that they liked aspects of the NeSC navigation because the values were visible. However there was also a feeling that this could be overwhelming at times, particularly on the homepage before the user had begun their search. This suggests that a middle ground between the collapsed (or accordian) facet navigation of the prototype and the fully expanded navigation of NeSC may be a realistic compromise. Further discussion on the pros and cons of facet navigation design can be read in the excellent blog post by James Kalbach.

Some users commented that they did not notice the facet navigation in the prototype because it did not immediately look like a faceted navigation system or because it’s design and position meant that users were more attracted to the results in the centre of the page. Currently the facets are closed by default and are styled to look like ordinary links. Although this accordian design of a facet can work, it requires additional features to communicate its purpose to the user. An arrow next to each facet is a common device used to indicate that it can be toggled to reveal the facet values. Additionally, expanding the first two facets and providing the rest closed is another strategy used demonstrate how the system works. It seems clear from the feedback during both phases of testing that additional design features or a different approach is required to make it easier for users to understand and successfully use the facet.

World Digital Library expanded facet example

However, if the prototype facet is open by default then the same issues may arise as was reported in the NeSC library. A compromise could be to limit the number of category values in each facet and provide a ‘Choose more…’ link so that users can expand the list if required. The University of Edinburgh Aquabrowser catalogue and World Digital Library are both examples of digital libraries using this feature in their faceted navigation systems, however each library implements the feature in quite different ways. Aquabrowser’s system is more user-friendly because it provides the full value list on a separate page and gives the user control over its presentation; relevance and alphabetical. World Digital Library expands the values within the facet, often with wordy facet values which are organised by relevance only (image). The list could clearly be difficult for users to navigate quickly and consequently may not enhance the feature. Indeed, this has already been witnessed while testing the NeSC digital library.

Another finding from the testing was the design and implementation of the combined facet navigation and breadcrumb system. I intend to discuss the feedback surrounding this in my next blog post.

Workpackage 2 (WP2) within the UX2 project plan focuses on usability and UX research through a variety of methods. Part of the WP2 plan includes a usability inspection of the National e-Science Centre (NeSC) digital library. In addition to the NeSC library interface, developments within WP4 (Concentration UX Enhancement) means that it has been possible to test a prototype system alongside NeSC library and make some comparisons. The differences between each system are essentially based on the version of Blacklight they are using; NeSC is a developed system based on an older version of Blacklight while the prototype uses the latest version, although it has not undergone any additional development yet.

An opportunity arose to conduct rapid user testing on both systems by tagging onto an existing usability training course run regularly at the university. Attendees were asked to participate at the end of the course for 15 mins. This Ambush-style Guerilla testing leant itself well to the current state of the prototype; giving participants one task that fitted the current capabilities of the system meant that longer usability testing was not appropriate yet. It also meant that feedback could be received relatively quickly and fed back into the user centred design process to be tested on the next group of participants two weeks later. To learn more about Guerilla testing (also referred to as discount usability) please check out these excellent resources:
The Least You Can Do by Steve Krug (video presentation)

As stated, the Ambush-style Guerilla testing was planned to coincide with the university’s usability training course. The first available courses were organised to take place on the 15th  and 30th June. The course itself is provided for those with little/no experience or knowledge of usability testing and typically has around 20 attendees. Recruiting participants in this way made it much easier to do at short notice. However, one downside to this method is that you don’t know anything about who you are testing and time limitations make it difficult to capture any demographic or profile information. Although the prototypes are not guaranteed to be tested on representative users, this method of testing does highlight issues during the early design stages of development prior to testing with representative users. This makes it a valuable exercise even if it is not exhaustive or scientific.

This blog post intends to layout the test plan created for the first round of testing which took place on the 15th June and provide a snapshot of some initial findings. The intention is to run the Guerilla tests again during the next training course on the 30th June using an updated version of the prototype.

Task scenarios

The task scenarios were constructed based on the data set used in each system. The full test plan was piloted on a member of staff before conducting the Guerilla tests.

Task 1a:

“As part of your work you need to read a selection of current presentations on particle physics. Using the prototype, can you find a suitable presentation published in the last 2 years?”

Task 1b:

“A lecturer has asked you to find the most recent presentation on Grid computing for an event coming up. Can you find this information using the prototype?”

Tasks 1 a and b were alternated between each participant to ensure one system did not benefit from familiarisation in the previous task. However, in total 3 participants completed the tasks leaving an odd number.  This was due in part to a delay at the start of the training course which meant it finished later than expected. People are understandably less likely to remain past the time when they would normally leave work. Consequently, time restrictions made it difficult to recruit additional participants. However, the next training course is a morning session and will likely have a better chance of recruiting more participants.


All three participants preferred to use the NeSC library as opposed to the prototype.

Participants used the facet navigation much less in the prototype than NeSC library.

The facet navigation in the prototype did not behave as a participant expected.  They also did not realise what it was at first.

A participant was surprised when NeSC library placed selections within bread crumb trail in a different order to that which they had selected.

Facets in the prototype don’t look like expandable sections or links.

Facets did not always cope presenting results when facets were selected in different order e.g. participants expected to find relevant presentations within the most recent date when narrowing by year first. However, this could be symptomatic of the small data set used for testing purposes and would need to be testing on a larger data set.

Facets with a large number of categories e.g. Subject, were difficult to scan in the NeSC library. For example, keywords such as Grid Computing featured in more than one subject category making it difficult for the user to know which category to select.

The facet navigation moves from left hand side to right hand side once a search has been conducted using the NeSC library. This was unexpected and confusing to one of the participants.

Often participants were unable to find results from the most recent year (2009), because it was not in the top ten of results presented in the facet of the prototype. The absence of a ‘more’ button made it difficult to search all years or most recent.

Participants often sought and expected years to be presented in a chronological order and often requested that this be provided to make searching easier. bookmarks

Twitter feed