(Un)reliable assessment : A case study.

Abstract

The drive towards quality assurance at South African universities, with 'consistency' of approach being one of its key features, has profound implications for assessment policies and practices in relation to equity. In this article we present a case study discussion of an investigation we undertook, as a department, into certain anomalies which arose in the assessment of a particular group of post-graduate students' research reports. We were puzzled by the variability in the marks awarded by three different markers of the same reports and set out to investigate what factors were producing this 'inter-marker [un]reliability'. Through a content and discourse analysis of the different assessors' written reports, we uncovered the implicit assessment categories and criteria which assessors were working with in their assessments. We discovered shared categories and criteria, as well as differences in how these were weighted. In the interests of equity and increased inter-marker reliability, we have developed a set of banded criteria on generic features of the research report which we intend to develop a set of banded criteria on generic features of the research report which we intend to trial. We also surfaced two unresolved issues: the use of language and the role of the writer's 'voice' in the research report. As a result of this investigation, we argue that the 'consistency' of assessment within and across universities aspired to by quality assurers (such as the HEQC in the South African context) is difficult to achieve and much still depends on professional judgement, intellectual position and personal taste.