Case 3: evaluating e-Learning Lab's website

In the evaluation of eLL’s website usability testing will be the approach when finding issues to approve on the website. Usability testing was the most used method for testing in the 1980’s and is still a very popular method. Today methods as heuristic evaluation and field studies are also dominant when designing and evaluating software ( Rogers et. al. 2002, p. 341). In usability testing typical users are tested under controlled conditions where they are given a set of predefined tasks that they have to complete. Users performance are for example measured in time spend on a task, number of errors or clicks. Most often users are given a questionnaire or are interviewed after the test (ibid.).

One of the characteristics of usability testing is that it is controlled by the evaluator and takes place under laboratory conditions where there are no disturbing elements. The user is recorded when doing the task and the video is used for later analysis.  User performance is then quantified and put up in tables or usability specifications that are used for evaluating the software. Even though data is quantified the number of users are too small to do a statistical analysis (ibid. p. 342).

Considerations about our approach

A script that shortly introduces the test should be made and presented for each user so that all the users are carried through the same procedure and differences are eliminated (ibid. 434).

Ideas for introduction script:

·      Tak fordi du vil deltage I usability test af eLL’s website

·      Data fra testen vil blive brugt til at evaluere og forbedre websiden

·      Der vil blive udarbejdet en rapport, som kun bliver delt inden for AAU’s “fire vægge”

·      Du får en række opgaver, som vi vil bede dig løse på hjemmesiden

·      Vi skal bede dig tænke højt mens du udfører opgaverne og testlederen vil stille spørgsmål undervejs

·      Du skal ikke skynde dig igennem opgaverne, bare tag dig den tid du har brug for og sig til, hvis du ikke kan løse en opgave, så går vi videre til den næste

·      Testen vil blive videofilmet og skærmen du udfører opgaverne på bliver ligeledes filmet, videoerne bliver ikke offentliggjort og du må være anonym I rapporten såfremt du ønsker det

·      Vi skal påpege at vi ikke ønsker at teste dine færdigheder inden for IT, men udelukkende hvor brugervenlig web siden er

·      Deltageren får evt. en samtykkeerklæring som han/hun skal underskrive

Rogers et al. also recommend letting the participants start with an explorations task to get familiar with the web site (ibid. p. 435). We have decided that the test should last maximum 30 min. so that we would not take too much of the participants’ time and thereby make it easier to find participants. With the limited time we will not spend too much time on an exploration task. Besides that the users chosen for the usability test are all familiar with eLL’s website or at least eLL as organisation. Instead of at exploration task we will start with an easy task that is loosely defined and very much up to the user how to complete.

According to Rogers et. al. a script for the evaluator to use during the test should also be carried out (ibid.). By following the script an even procedure for all the participants are ensured. Besides that the evaluator are reminded to stay within the time schedule so that the participant does not spend too much time on one task. An estimated amount of time for each task has to be decided by the evaluators before the test.

Ideas for evaluator script:

·      Hvad tænker du om X, når du klikker på X, når du ser X?

·      Kan du fortælle mig hvad du tænker nu?

·      Er du gået I stå?

·      (Hvis opgaven overskrider den estimerede tid:) Jeg bliver desværre nødt til at bede dig om at gå videre til næste opgave.

·      Evaluatoren skal være opmærksom på følgende:

o   Yd først assistance til en opgave når deltageren har gjort adskillige forsøg selv og er gået helt I stå

o   Giv ikke hjælpende hints undervejs

o   Mere?

When making the specific tasks for the test Morville & Rosenfeld have some recommendations for the evaluator (2007, p. 260). Inspired by these recommendations we will, as mentioned, let the participants start out with an open “task”. In relation to that Morville & Rosenfeld recommends to make some topical questions and not only specific tasks (ibid.). Even though the usability test is an artificial setting the evaluator should build in some realistic tasks (ibid.). In the evaluation of eLL’s web site it could be necessary to make specific task for the specific user group taking this recommendation into consideration.