Posts Tagged ‘methodologies’
Sampling bias in online surveys
Posted on: March 5, 2010
- In: Evaluation
- 3 Comments
Bias is an issue that anyone gathering user data is weary of. Whether its usability testing, face-to-face interviews or online questionnaires, bias can affect the strength and integrity of results in a variety of ways. Question design is one of the most influential factors should therefore be given careful consideration. Leading questions can inadvertently give participants an idea of the desired answer and influence their response. However, sampling bias can also have a significant affect on the research results and is often overlooked by researchers.
I was reading Europeana’s Annual Report this week and noticed that the results from their online visitor survey was on the whole very positive. Reading the survey report in more detail I realised it was possible that sample bias may be affecting the survey results. Data from online visitor surveys are normally gathered using an intercept which invites a visitor to participate in the research when they arrive to the site. Anyone visiting the site is who receives this invite is eligible to participate making them ‘self-selected’. This means that they decide to participate, not the researcher. Their motivation for participating may be related to the topic of the survey or the incentive provided to garner interest. Consequently their participation is unlikely to provide a repserentative sample.
For example, those who participated in Europeana’s survey are more likey to be motivated by their enthusiasm and interest in the website. Certainly those who are apathetic or indifferent to the website are less likely to have participated. This is supported by the proportion of participants who were regular visitors to the site. Only 8.6% of participants were first time visitors and the results from these participants was generally more positive than the participants who had visited the site before. It would be interesting to find out if a larger sample of first time users would alter these results.
So what can researchers do to prevent sample bias in their results? It is very difficult to completely remove sample bias especially in online surveys where the researcher has no control over who participates. Generally speaking visitor surveys will always carry the risk of bias so the aims of the survey should take this into account. Designing a mixture of open and closed questions will provide some insight into the participant’s motivation. Descriptive answers which require more thought are less likely to be fully answered by those motivated by the incentive. It also provides the added benefit of giving users the opportunity to provide their own feedback. It is interesting to note the Europeana did not do this, leading some participants to email their comments to the researchers. Providing an optional section at the end of the survey for final comments could have provided rich feedback not obtained through closed questions. Indeed the comments Europeana received often described situations where users’ had trouble using the site or disliked a particular design feature.
Avoid asking questions which relate to the user’s overall opinion of the system before they have used all the features as it will not provide accurate results. For example, 67% of users stated they had never used the “My Europeana” feature before and were therefore unable to provide feedback on it. Usability testing often provides more insight into these issues by gathering this information retrospectively after asking a user to carry out tasks using the site. If it’s possible to use survey software which can do this then it is recommended because it is more likely to gather meaningful results. It is only after trying to complete a task that a user will be able to accurately describe their experience.
It is worth noting that Europeana have also conducted user testing with eyetracking in addition to focus groups and expert evaluations. The results of these are due to be published soon and I look forward to reading them. It will be interesting to compare the results against our heuristic inspection of Europeana and other DLs.
- In: UX2
- Leave a Comment
My last post said that an explanation of each methodology would be provided for those unfamiliar with the terms. As promised, details are provided below:
Heuristic Evaluation
Heuristic evaluations are well documented on the web with many authoritative resources. Instead of paraphrasing other sources I thought it would be more appropriate to provide links.
Brief definition by Usability First
Jakob Nielsen’s ‘How to’ paper provides a comprehensive guide to the methodology from when it was originally developed: http://www.useit.com/papers/heuristic/heuristic_evaluation.html
Sitepoint’s step-by-step guide provides a more detailed explanation of each heuristic which is straighforward and easy to understand: http://www.sitepoint.com/article/heuristic-evaluation-guide/
Usability.gov outlines the original heuristics in addition to subsequent variations: http://www.usability.gov/methods/heuristiceval.html
A detailed check-list of every heuristic for those conducting a heuristic evaluation: http://www.stcsig.org/usability/topics/articles/he-checklist.html
The benefits of a heuristic evaluation from Usability Net: http://www.usabilitynet.org/tools/expertheuristic.htm
Cognitive Walk-through
Short description and definition
Comprehensive outline of the method: http://www.tiresias.org/tools/cognitive_walkthrough.htm
Defining the procedure of a walk-through: http://www.pages.drexel.edu/~zwz22/CognWalk.htm
Some of the characteristics of a cognitive walk-through include the creation of personas and the detailed documentation of tasks step-by-step in order to pinpoint exactly where the interface might fail. As with a heuristic evaluation, more than one evaluator is ideal however a cognitive walk-through often includes developers in the team of evaluators as well as usability experts.
Relationship with UX2.0 project
Sitepoint’s guide suggests conducting a heuristic evaluation by using one of three approaches:
- Develop a set of tasks and ask your evaluators to carry them out.
- Provide evaluators with the goals of the system, and allow them to develop their own tasks.
- Ask evaluators to assess your dialogue elements.
As discussed in the previous post, the evaluation intends to follow the set of heuristics while also carrying out typical user tasks (approach 1). Therefore the evaluation will predominantly be a heuristic evaluation and not a cognitive walk-through.
The main user task of a digital library is searching for a piece of information. Specific details of the information a user might be searching for will vary according to each DL and cannot be fixed. As a result, specific details of each task will be appropriate to each site.
UX 2.0 Usability planning
Posted on: September 4, 2009
- In: inf11 | UX2
- 3 Comments
So the first part of the usability research taking place in the UX 2.0 project is to perform a usability inspection of selected digital libraries (DLs). In order to do this two things had to be decided:
- What DLs to inspect
- How to perform the inspection
In this entry I have mapped out how these decisions were made and the implications.
The most difficult thing about selecting DLs to inspect was in narrowing the list because there are so many DLs out there. How many will give a sufficient breadth of information for comparison? What criteria should be used and which should be excluded? These were all questions that the team had to answer. As we wanted to compare our findings with the evaluation of the digital library, library@nesc later on in the project, it seemed appropriate to exclude commercial publisher digital libraries and focus on public digital libraries. In addition to that, the findings from the WorldCat usability testing report published for the ALA Annual in July 2009 revealed that academic users favour searching local, national and worldwide collections together as opposed to public library patrons who are interested in resources which are geographically close. This led us to think in terms of the geographic reach of DLs and the differences between them. As a result we selected 5 digital libraries which represent each geographic location; worldwide, continental, nationwide, regional and local. From this the following DLs were selected:
| Attribute | Digital Library | Web address |
| Worldwide | World Digital Library | http://www.wdl.org/en/ |
| Continental (Europe) | Europeana | http://www.europeana.eu |
| Nationwide (UK) | British Library | http://www.bl.uk |
| Regional (Scotland) | SCRAN | http://www.scran.ac.uk/ |
| Local (Edinburgh) | Edinburgh University Aqua Browser | http://aquabrowser.lib.ed.ac.uk/ |
Next thing to do was decide how to conduct the inspection. There are a number of well known and commonly used usability methodologies available. A number of factors affecting the scope of this inspection helped to narrow the choice:
- Scope: the inspection was proposed as a quick appraisal of the current state of digital libraries and was not intended as a detailed evaluation.
- Time-scales: the short time-scale meant that the inspection had to be done quickly. As a result, user testing would not be achievable at this stage in the project
Consequently, it would not be possible to evaluate the usefulness of each DL as outlined by Tsakonas and Papatheodorou (2007) and their triptych methodology. Factors such as the relevance, format, reliability and coverage would not be examined at this time. Instead the focus would be on the usability of the system to the user such as ease of use, aesthetics, navigation, terminology and learnability. As digital libraries generally have a well developed strategy and scope, it is more important to focus attention on the structure, skeleton and surface of the DL as explained by Jesse James Garrett in his book ‘The Elements of User Experience’. This includes information architecture, navigation design and visual design such as page format, colours and typography.
With all this in mind it was decided that a heuristic evaluation would be suitable. However, co-creator of these heuristics, Jakob Nielsen points out that heuristic evaluations are better when carried out by more than one evaluator. As there are no other specialists working on this project this would not be possible. However, as this inspection is intended as a quick evaluation of current DLs it was not considered detrimental to the research. To try and limit this issue, use of the cognitive walk-through method will also be integrated into the inspection. Formal task scenarios will not be created but typical user tasks such as searching will be considered when evaluating each DL. It is hoped that doing so will highlight any barriers to task success when its not possible to test with actual users.
For anyone who is unsure what a heuristic evaluation and cognitive walk-through entail, I plan to explain these in my next blog post.
So after deciding on the digital libraries to inspect and the method to inspect them, I am now at the stage of analysing each library and collecting my findings. Every usability expert has their own method for doing this but I find that familiarising myself with each site first then jotting down brief notes on each issue accompanied by a screen grab works best for me. After that, issues will be written up in detail assigned a severity rating and discussed. In addition, positive findings and the development of collaborative or personalised systems (if any) will also be examined. Finally, each DL will be compared and contrasted and conclusions drawn.
I hope this has helped to provide insight into the early stages of the usability research taking place. Please feel free to comment or discuss any aspect of the methodology.
Researching Usability 