Instrument validation and evaluation of problem-based learning tutorial performance of undergraduate nursing students

Abstract

PURPOSE: The purpose of this two-phased study was to determine the performance of undergraduate nursing students in problem-based learning (PBL) tutorials using a validated evaluation instrument. RELEVANCE: to determine the effectiveness of the PBL learning approach relative to the South African student. Phase 1 led to the validation of an instrument and Phase 2 evaluated the performance of nursing students in PBL tutorials using the validated instrument. PARTICIPANTS: Phase 1 participants included academic experts (n=8) selected by means of purposive, maximum variation sampling. Phase 2 participants included the total population of undergraduate nursing students (n=53) and facilitators (n=6). METHODS: A quantitative research approach was used to inform the overarching design that was descriptive and comparative in Phase 2 of the study. Phase 1 employed statistical techniques for instrument validation and refinement. Phase 1 data were collected in three round of a Delphi survey. After completion of the first two rounds a rating instrument with a 4-point (0-3) rating scale was developed referred to as the Tutorial Performance Rating Instrument. During the third and final round of the Delphi survey ‘weighting’ of each main-item, sub-items and the rating scale took place, incorporating the Subjective Judgement Model using pair-wise comparisons on linear visual analogue scales. ANALYSIS: Relative weights were determined and following statistical analysis ratio scales were developed creating a unique ‘weight’ to each item and the rating scale. This ‘weight’ was represented in a percentage allowing each main-item construct and each sub-item to be placed in a hierarchy from highest to lowest percentage. Calculation on a student assessment would become time consuming and subject to error if done manually. A computer-based program referred to as the Tutorial Performance Evaluator was developed to carry out all the calculations of the percentages allocated to the items and rating scale. A replica of the Tutorial Performance Rating Instrument was built into the programme. In Phase 2 of the study a self-assessment and facilitator-assessment on each student’s tutorial performance first-to fourth-year was carried out using the Tutorial Performance Rating Instrument. Following a one on one meeting between the student and the facilitator consensus agreement was reached on an acceptable rating against each item on the evaluation instrument. The latter was entered into the computer and a percentage for each main-item construct and a total percentage was calculated efficiently and accurately in 20 seconds. RESULTS: The results showed that first-year students struggled in all aspects of the PBL tutorial. Furthermore they did not possess the skills required for self-assessment. A small group of the second-year students struggled in the tutorials but were better able to carry out a self-assessment. The third-year students showed a slight drop in score when compared with the second-year students and this could be attributed to the new subjects introduced in the third-year of study. There was a substantial improvement in the results achieved by the fourth-year students in all the aspects of the PBL tutorial and showed a good correlation in carrying out a self-assessment when compared with the facilitator assessment. iv RECOMMENDATIONS: This was a cross-sectional study and a longitudinal study should be carried out in future research to assess the individual’s progress from first-to fourth-year in the PBL approach to learning. Greater academic support should be given to first-year students or alternatively the ‘at risk’ students should be given a foundation course to assist them with communication skills and learning skills.