Mental models : have users' mental models of search engines improved in the last ten years?
Date
2011-05-26
Authors
Mlilo, Sifiso
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Mental models can be defined as mental representations or cognitive mappings of external
systems that allow for effective descriptions, explanations and predictions of those systems
(Staggers & Norcio, 1993). The quality of users’ mental models of search engines has been
investigated in a number of studies across a range of disciplines. The results have been uniform
in concluding that users’ mental models of search engines are inaccurate and incomplete (Jansen
& Spink, 2006; Muramatsu & Pratt, 2001; Thatcher & Greyling, 2003). This conclusion has
corroborated studies proposing that users show poor search performance (Muramatsu & Pratt,
2001). This study investigated the accuracy and completeness of mental models users have of
search engines in the context of a comparison of matched data obtained from samples from 2000
and 2010. The same methodology employed in the study conducted on the 2000 sample was
used. Eighty volunteer participants were asked to complete a maximum of two computer-based
tasks and onscreen capture software was used to record their search actions (time, steps, etc.).
The recordings were then played back to the participants and a retrospective protocol interview
was conducted to determine the reasoning behind their actions. They were also asked to provide
an illustration of how they thought a search engine worked. A short satisfaction and confidence
questionnaire was also administered. Statistical significance was found for the performance
measures of time and steps. The differences were quite large with the 2010 sample using less
than half the time used by the 2000 sample and almost half the number of steps. The
performance was still significant after controlling for experience. The chi-squared test conducted
for the performance measure accuracy also revealed statistical significance with the 2010 sample
showing significantly greater accuracy. The chi-squared comparison of the salient features
indicated that seven of the seventeen features were more frequent at Time 2. Seven of the salient
features were more frequent at Time 1. Where the cluster analysis of the 2000 sample revealed
four distinct clusters, three clusters emerged from the 2010 sample. It was clear from the
analyses conducted that, even if not to a large degree, the accuracy and completeness of users’
mental models of search engines had demonstrably improved in the last decade. So while users’
mental models of search engines remain inaccurate and incomplete, their alignment with
designer’s conceptualisations has improved.