Researching Usability

UX2.0 Guerilla usability tests

Posted on: June 23, 2010

Workpackage 2 (WP2) within the UX2 project plan focuses on usability and UX research through a variety of methods. Part of the WP2 plan includes a usability inspection of the National e-Science Centre (NeSC) digital library. In addition to the NeSC library interface, developments within WP4 (Concentration UX Enhancement) means that it has been possible to test a prototype system alongside NeSC library and make some comparisons. The differences between each system are essentially based on the version of Blacklight they are using; NeSC is a developed system based on an older version of Blacklight while the prototype uses the latest version, although it has not undergone any additional development yet.

An opportunity arose to conduct rapid user testing on both systems by tagging onto an existing usability training course run regularly at the university. Attendees were asked to participate at the end of the course for 15 mins. This Ambush-style Guerilla testing leant itself well to the current state of the prototype; giving participants one task that fitted the current capabilities of the system meant that longer usability testing was not appropriate yet. It also meant that feedback could be received relatively quickly and fed back into the user centred design process to be tested on the next group of participants two weeks later. To learn more about Guerilla testing (also referred to as discount usability) please check out these excellent resources:
The Least You Can Do by Steve Krug (video presentation)

As stated, the Ambush-style Guerilla testing was planned to coincide with the university’s usability training course. The first available courses were organised to take place on the 15th  and 30th June. The course itself is provided for those with little/no experience or knowledge of usability testing and typically has around 20 attendees. Recruiting participants in this way made it much easier to do at short notice. However, one downside to this method is that you don’t know anything about who you are testing and time limitations make it difficult to capture any demographic or profile information. Although the prototypes are not guaranteed to be tested on representative users, this method of testing does highlight issues during the early design stages of development prior to testing with representative users. This makes it a valuable exercise even if it is not exhaustive or scientific.

This blog post intends to layout the test plan created for the first round of testing which took place on the 15th June and provide a snapshot of some initial findings. The intention is to run the Guerilla tests again during the next training course on the 30th June using an updated version of the prototype.

Task scenarios

The task scenarios were constructed based on the data set used in each system. The full test plan was piloted on a member of staff before conducting the Guerilla tests.

Task 1a:

“As part of your work you need to read a selection of current presentations on particle physics. Using the prototype, can you find a suitable presentation published in the last 2 years?”

Task 1b:

“A lecturer has asked you to find the most recent presentation on Grid computing for an event coming up. Can you find this information using the prototype?”

Tasks 1 a and b were alternated between each participant to ensure one system did not benefit from familiarisation in the previous task. However, in total 3 participants completed the tasks leaving an odd number.  This was due in part to a delay at the start of the training course which meant it finished later than expected. People are understandably less likely to remain past the time when they would normally leave work. Consequently, time restrictions made it difficult to recruit additional participants. However, the next training course is a morning session and will likely have a better chance of recruiting more participants.


All three participants preferred to use the NeSC library as opposed to the prototype.

Participants used the facet navigation much less in the prototype than NeSC library.

The facet navigation in the prototype did not behave as a participant expected.  They also did not realise what it was at first.

A participant was surprised when NeSC library placed selections within bread crumb trail in a different order to that which they had selected.

Facets in the prototype don’t look like expandable sections or links.

Facets did not always cope presenting results when facets were selected in different order e.g. participants expected to find relevant presentations within the most recent date when narrowing by year first. However, this could be symptomatic of the small data set used for testing purposes and would need to be testing on a larger data set.

Facets with a large number of categories e.g. Subject, were difficult to scan in the NeSC library. For example, keywords such as Grid Computing featured in more than one subject category making it difficult for the user to know which category to select.

The facet navigation moves from left hand side to right hand side once a search has been conducted using the NeSC library. This was unexpected and confusing to one of the participants.

Often participants were unable to find results from the most recent year (2009), because it was not in the top ten of results presented in the facet of the prototype. The absence of a ‘more’ button made it difficult to search all years or most recent.

Participants often sought and expected years to be presented in a chronological order and often requested that this be provided to make searching easier.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s bookmarks

Twitter feed


%d bloggers like this: