1 EVIDENCE OF DETERMINING CLINICAL UTILITY IN THE DEVELOPMENT OF ASSESSMENTS IN OCCUPATIONAL THERAPY: A SCOPING REVIEW. LINDSAY DAWSON 2272561 A Research Report submitted to the Department of Occupational Therapy of the University of the Witwatersrand, Johannesburg, in partial fulfilment of the requirements for the degree of Masters of Science in Occupational Therapy, Johannesburg, June 2022. i Declaration I, Lindsay Dawson declare that this Research Report is my own, unaided work. It is being submitted for the Master of Sciences in Occupational Therapy degree at the University of Witwatersrand, Johannesburg. It has not been submitted before for any degree or examination at any other University. Lindsay Dawson _________________ Name of candidate Signature of candidate 23rd day of June 2022 in Centurion. ii Plagiarism Declaration Faculty of Health Sciences, Postgraduate Office Phillip V Tobias Building, 2nd Floor Cnr York & Princess of Wales Terrace, Parktown 2193 Tel: (011) 717 2745 | Fax: (011) 717 2119 Email: Mathoto.senamela@wits.ac.za PLAGIARISM DECLARATION TO BE SIGNED BY ALL HIGHER DEGREE STUDENTS SENATE PLAGIARISM POLICY: APPENDIX ONE I Lindsay Dawson (Student number: 2272561) am a student registered for the degree of MSc Occupational Therapy in the academic year 2019. I hereby declare the following: • I am aware that plagiarism (the use of someone else’s work without their permission and/or without acknowledging the original source) is wrong. • I confirm that the work submitted for assessment for the above degree is my own unaided work except where I have explicitly indicated otherwise. • I have followed the required conventions in referencing the thoughts and ideas of others. • I understand that the University of the Witwatersrand may take disciplinary action against me if there is a belief that this is not my own unaided work or that I have failed to acknowledge the source of the ideas or words in my writing. • I have included as an appendix a report from “Turnitin” (or other approved plagiarism detection) software indicating the level of plagiarism in my research document. Signature: Date: 19/01/2022 iii Acknowledgements I would like to express my sincere gratitude and appreciation for everyone who has supported me through the process of completing this degree. • To my family and fiancé, without your love, patience, support, and encouragement, this would not have been possible. • To my supervisor, Dr Janine van der Linde, thank you for your assistance and guidance during the completion of the study. iv Abstract Occupational therapists rely on information gained from standardised assessments, screening instruments, and clinical observations to determine a client’s strengths and weaknesses (American Occupational Therapy Association [AOTA], 2020; Brown, 2010; Foxcroft & Roodt, 2013; Kramer et al., 2009; Richardson, 2010). The information gained from the assessment will be used for planning of intervention and discharge, writing of reports and documentation, as well as applying for funding from medical aids (Alotaibi et al., 2009; Asaba et al., 2017; Benson & Clark, 1982; Scott et al., 2006; Unsworth, 2000). Clinical practice is therefore demanding assessments that are not only psychometrically sound but are appropriate to the context in which they are used (Bossuyt et al., 2012; Foxcroft et al., 2004; Glover & Albers, 2007; Nalder et al., 2017). Clinical utility can be described as how useful and appropriate the assessment is within a specific setting (Benson & Clark, 1982; Lesko et al., 2010; Macy, 2012; Nalder et al., 2017). This is especially important within the South African context, where occupational therapists often use assessments that were developed in the Global North. During assessment development, validity and reliability are often the only factors determined to establish the psychometric properties of an assessment (Benson & Clark, 1982; Salmond, 2008; Switzer et al., 1999). However, it is essential to determine the clinical utility of an assessment during the psychometric testing phase in assessment development. This will allow the developer to adjust the assessment to suit the characteristics and needs of the population and context (Bowyer et al., 2012). A scoping review was conducted to determine whether there is evidence of clinical utility being determined as part of psychometric testing during assessment development in occupational therapy. The study aimed to map the available literature on clinical utility in occupational therapy and identify possible gaps within the existing literature. A comprehensive search of published and unpublished literature between January 2005 and December 2020 was conducted. The search yielded 38 applicable studies. The results of the study indicated an increase in the number of studies done on the clinical utility of assessments within the last 10 years. The majority of the studies were conducted in Global North countries including Australia, the United States of America, and the United Kingdom. The included studies used a variety of study designs, including qualitative, quantitative, and mixed method research designs. The study also v included both systematic and literature reviews. However, these designs often did not include the context for which the clinical utility was being determined. A wide variety of components of clinical utility were included within the research, emphasising the lack of a standardised definition as well as a process for establishing the clinical utility. Limited studies on clinical utility were conducted during the assessment development process. Studies were mostly conducted when an already developed assessment was used in a different context to which it was initially developed. None of the included research was conducted in South Africa. Therefore, the results from the Global North cannot be generalised to the diverse context and rich cultural population of South Africa, which forms part of the Global South. vi TABLE OF CONTENTS Declaration ................................................................................................................ i Plagiarism Declaration ............................................................................................. ii Acknowledgements ................................................................................................. iii Abstract................................................................................................................... iv List of Tables .......................................................................................................... ix List of Figures .......................................................................................................... x List of Appendices .................................................................................................. xi Operational Definitions .......................................................................................... xii List of Abbreviations .............................................................................................. xv Chapter 1 : Introduction ............................................................................................ 1 1.1 Introduction & background ................................................................................ 1 1.2 Problem statement............................................................................................ 2 1.3 Research question ............................................................................................ 3 1.4 Research aims .................................................................................................. 3 1.5 Research objectives ......................................................................................... 3 1.6 Significance of the study ................................................................................... 3 1.7 Outline of this research report .......................................................................... 4 Chapter 2 : Literature Review ................................................................................... 5 2.1 Introduction ....................................................................................................... 5 2.2 The role of assessment in occupational therapy ............................................... 5 2.2.1 Occupational therapy ............................................................................... 5 2.2.2 Assessments in occupational therapy ...................................................... 7 2.3 Development of contextually appropriate assessments .................................... 8 2.3.1 Assessment development ........................................................................ 8 2.3.2 Psychometric testing in assessment development ................................ 10 2.3.3 Contextual considerations in assessment development ........................ 13 2.4 Clinical utility ................................................................................................... 16 2.4.1 Clinical utility as a concept ..................................................................... 17 2.4.2 Clinical utility in assessment development ............................................. 17 2.4.3 Clinical utility in therapeutic and rehabilitative healthcare services ........ 19 2.5 Clinical utility in occupational therapy ............................................................. 21 vii 2.6 Summary ........................................................................................................ 23 Chapter 3 : Methodology ........................................................................................ 25 3.1 Introduction ..................................................................................................... 25 3.2 Study design ................................................................................................... 25 3.3 Inclusion and exclusion criteria ....................................................................... 26 3.3.1 Inclusion criteria ..................................................................................... 26 3.3.2 Exclusion criteria .................................................................................... 26 3.4 Data collection procedure ............................................................................... 27 3.4.1 Search strategy ...................................................................................... 27 3.4.2 Summary of data collection ................................................................... 29 3.5 Data extraction ............................................................................................... 30 3.6 Data analysis and synthesis ........................................................................... 30 3.7 Ethical considerations ..................................................................................... 32 3.8 Conclusion ...................................................................................................... 32 Chapter 4 : Results ................................................................................................. 33 4.1 Introduction ..................................................................................................... 33 4.2 Distribution of studies ..................................................................................... 33 4.2.1 Distribution of articles by year of publication .......................................... 33 4.2.2 Distribution of articles by country ........................................................... 34 4.3 Research design, methodology, sample size and population of included articles 34 4.3.1 Research design and methodology ....................................................... 34 4.3.2 Sample size ........................................................................................... 35 4.3.3 Population .............................................................................................. 36 4.4 Clinical utility investigation within occupational therapy .................................. 36 4.4.1 Domains of occupational therapy included in clinical utility studies ....... 36 4.4.2 Inclusion of clinical utility within assessment development .................... 37 4.4.3 Components of clinical utility ................................................................. 38 4.5 Conclusion ...................................................................................................... 41 Chapter 5 : Discussion ........................................................................................... 43 viii 5.1 Introduction ..................................................................................................... 43 5.2 Demographics, research design, sample size and population ........................ 43 5.2.1 Demographics of the included literature ................................................ 43 5.2.2 Research design and methodology used ............................................... 44 5.2.3 Sample sizes ......................................................................................... 46 5.2.4 Population .............................................................................................. 46 5.3 Clinical utility within occupational therapy ....................................................... 47 5.3.1 Clinical utility as a concept in occupational therapy ............................... 47 5.3.2 Domains in occupational therapy assessments included in clinical utility studies 51 5.3.3 Evidence of clinical utility studies in assessment development ............. 52 5.4 Conclusion ...................................................................................................... 53 Chapter 6 : Conclusion and Recommendations ..................................................... 55 6.1 Conclusion ...................................................................................................... 55 6.2 Implications for practice .................................................................................. 57 6.3 Recommendation for future research ............................................................. 58 6.4 Limitations of this study .................................................................................. 59 6.4.1 Limitations of the included studies ......................................................... 60 6.5 Conflict of interest ........................................................................................... 60 6.6 Funding ........................................................................................................... 60 References ................................................................................................................ 61 Appendices................................................................................................................ 72 ix List of Tables Table 3.1 Search terms for Step two of data collection…………………………………27 Table 3.2 Articles found per database……………………………………………………29 Table 4.1 Average sample size per research design……………………………………36 Table 4.2 Components of clinical utility assessed……………………………………….39 x List of Figures Figure 1.1 Outline of the research report…………………………………………………...4 Figure 2.1 The domain of occupational therapy according to the Occupational Therapy Practice Framework: 4rd edition (AOTA, 2020: p.7)……………………………………….6 Figure 2.2 The proposed steps to determine clinical utility alongside the psychometric properties of an assessment tool (Bowyer et al., 2012: p.21)………………………….18 Figure 3.1 PRISMA flow diagram (Page et al., 2021: p.5)……………………………...30 Figure 4.1 Distribution of articles by year of publication…………………………………33 Figure 4.2 Distribution of articles by country……………………………………………...34 Figure 4.3 Distribution of articles by research design……………………………………35 Figure 4.4 Distribution of articles by domains of occupational therapy……………….37 xi List of Appendices Appendix A: Record of literature searches……………………………………………….75 Appendix B: Data extraction tables……………………………………………………….76 Appendix C: Ethical waiver…...……………………………………………………………93 Appendix D: Turnitin report………………………………………………………………...94 xii Operational Definitions Assessment A tool or instrument that is used to describe a client’s current functioning in terms of their occupational profile, client factors, performance skills, performance patterns and environmental factors (AOTA, 2020) Client factors “Specific capacities, characteristics, or beliefs that reside within the person and that influence performance in occupations. Client factors include values, beliefs, and spirituality; body functions; and body structures” (AOTA, 2020: p. 75) Clinical utility Clinical utility can be described as how useful and appropriate the assessment is within a specific setting (Benson & Clark, 1982; Lesko et al., 2010; Macy, 2012; Nalder et al., 2017). Context “Construct that constitutes the complete makeup of a person’s life as well as the common and divergent factors that characterize groups and populations. Context includes environmental factors and personal factors.” (AOTA, 2020: p. 76) Environmental factors “Aspects of the physical, social, and attitudinal surroundings in which people live and conduct their lives.” (AOTA, 2020: p. 76) Occupation “Everyday personalized activities that people do as individuals, in families, and with communities to occupy time and bring meaning and purpose to life. Occupations can involve the execution of multiple activities for completion and can result in various outcomes. The broad range of occupations is xiii categorized as activities of daily living, instrumental activities of daily living, health management, rest and sleep, education, work, play, leisure, and social participation.” (AOTA, 2020: p. 79) Occupational performance “Accomplishment of the selected occupation resulting from the dynamic transaction among the client, their context, and the occupation.” (AOTA, 2020: p. 80) Occupational profile “Summary of the client’s occupational history and experiences, patterns of daily living, interests, values, needs, and relevant contexts” (AOTA, 2020: p. 80) Performance patterns “Habits, routines, roles, and rituals that may be associated with different lifestyles and used in the process of engaging in occupations or activities. These patterns are influenced by context and time and can support or hinder occupational performance.” (AOTA, 2020: p. 80) Performance skills “Observable, goal-directed actions that result in a client’s quality of performing desired occupations. Skills are supported by the context in which the performance occurred and by underlying client factors.” (AOTA, 2020: p. 80) Psychometric properties “The properties of the instrument as it functions within the context.” (Switzer et al., 1999: p.399). This provides information about how accurate the assessment tool is and the quality of results that can be expected from the assessment (De Souza et al., 2017; Salmond, 2008). This construct usually only included the reliability and validity of assessments (Bowyer et al., 2012; Switzer et al., 1999). xiv Reliability Reliability refers to the ability of the assessment tool to provide a true score of the constructs being assessed and that the scoring is not altered by the examiner or environmental conditions (Salmond, 2008; Switzer et al., 1999). Validity The validity of an assessment tool refers to the ability of the tool to assess the construct for which it was developed (Salmond, 2008; Switzer et al., 1999). xv List of Abbreviations AOTA American Occupational Therapy Association CAP-M Comparative Analysis of Performance – Motor COPM Canadian Occupational Performance Measure CVI Content validity index DASH Disability of the Arm Shoulder and Hand FLACC The Face, Legs, Activity, Cry, Consolability HART Handicap Assessment and Resource Tool HIV Human immunodeficiency virus HPCSA Health Professions Council of South Africa I-HOPE In Home Occupational Performance Evaluation IRO Inventory of Reading Occupations JBI Joanna Briggs Institute MATCH-ACES Matching Assistive Technology and Child with augmentative communication evaluation simplified supplement MET Multiple Errands Test NICU Neonatal Intensive Care Unit PAT Pain Assessment Tool PICU Paediatric Intensive Care Unit PPR profile Pleasure, Productivity and Restoration Profile PRISMA Preferred Reporting Items for Systematic reviews and Meta-Analyses PRISMA-ScR Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews xvi SARA Self-Assessment of Role-performance and activities of daily living Abilities SASISI South African Sensory Integration Screening Instrument SCOPE Short Child Occupational Profile SDO Satisfaction with Daily Occupations SEQ Sensory Experiences Questionnaire SIPT Sensory Integration and Praxis Tests SP Sensory Profile SPM Sensory Processing Measure UK United Kingdom USA United States of America WRI-S Worker Role Interview Swedish version 1 CHAPTER 1 : INTRODUCTION 1.1 Introduction & background As part of the therapeutic process, occupational therapists rely on information gained from standardised assessments, screening instruments, and clinical observations to determine a client’s strengths and weaknesses (American Occupational Therapy Association [AOTA], 2020; Brown, 2010; Foxcroft & Roodt, 2013; Kramer et al., 2009; Richardson, 2010). Clinical practice is therefore demanding assessments that are not only psychometrically sound but are appropriate to the context in which they are used (Bossuyt et al., 2012; Foxcroft et al., 2004; Glover & Albers, 2007; Nalder et al., 2017). During psychometric testing, determining the validity of an assessment should include determining the clinical utility and thus whether it is relevant to the context (Switzer et al., 1999). Even though no formal definition of clinical utility is available in the literature (Smart, 2006), clinical utility can be described as how useful and appropriate the assessment is within a specific setting (Benson & Clark, 1982; Lesko et al., 2010; Macy, 2012; Nalder et al., 2017). Occupational therapists often use internationally developed assessments and adapt them to the context and needs of the population (Pascoe & Norman, 2011). This could result in an assessment that is not appropriate to the culture, nor useful within the context (Alotaibi et al., 2009; Foxcroft, 2012; Foxcroft & Roodt, 2013). In South Africa, it is especially important to ensure that assessments are appropriate for our unique context. Occupational therapists in South Africa work in a variety of settings, including private and public hospitals and clinics, rehabilitation centres, insurance companies, nursing homes, care centres, and schools (Jansen van Vuuren et al., 2020). Services are provided to people of different ages, cultural backgrounds, and socio-economic standings (Jansen van Vuuren et al., 2020; van der Linde, 2019; Van Stormbroek & Buchanan, 2017). Research in determining and/or investigating the clinical utility of assessments has been emerging within therapeutic and rehabilitative healthcare services over the last 20 years (Burton & Tyson, 2015; Higa et al., 2002; Nalder et al., 2017; Tyson & Connell, 2009a; Voepel-Lewis et al., 2008). It is therefore important and relevant to determine the clinical utility in the development of assessments in the field of occupational therapy. 2 1.2 Problem statement Assessments are often developed internationally and used within different contexts, resulting in assessments that are not contextually appropriate or suited for the culturally diverse population (Foxcroft, 2012; Pascoe & Norman, 2011). Bowyer et al. (2012) highlighted a limitation during traditional assessment development. They noted that only the validity and reliability of new assessments are determined, and the test developer does not investigate the clinical usefulness of the assessment for health care practitioners specific to their context. Using an assessment in a context that is different from the initial context for which it was developed, impacts the accuracy and usefulness of the results (Jorquera-Cabrera et al., 2017; Van Jaarsveld et al., 2012; Wren & Benson, 2004). Measurable and reliable outcomes are required to determine the effectiveness of treatment in all settings, motivate for funding, and emphasise the need for occupational therapy services in different contexts (Alotaibi et al., 2009; Asaba et al., 2017; Benson & Clark, 1982; Rudman & Hannah, 1998; Scott et al., 2006; Unsworth, 2000). It is thus essential for occupational therapists to select an assessment that is appropriate for their therapeutic goals, population, and setting (Darzins et al., 2016; Rudman & Hannah, 1998; Unsworth, 2000). Limited studies have been done to determine clinical utility in not only therapeutic and rehabilitative healthcare services (Bowyer et al., 2012), but also in the field of occupational therapy. Consequently, occupational therapists from the Global South, use assessments that have been developed in the Global North, resulting in contextually inappropriate assessments (Pascoe & Norman, 2011). In a country with a multicultural population and diverse contexts such as South Africa, it is essential to select contextually appropriate assessments to provide a service that is fair and just (Health Professions Council of South Africa [HPCSA], 2008). Using assessments that are not contextually appropriate will impact not only clinical practice but also research (Brown, 2010). The use of inappropriate assessments may result in clients not receiving the appropriate services, lack of funding for therapeutic intervention and poor record keeping. Research containing data from inappropriate assessments will contain questionable results (Brown, 2010), impacting the usefulness of the research in evidence-based practice. It is thus important to determine whether research into the clinical utility of assessments is conducted during the assessment development process, for the context in which the assessment is intended to be used. 3 1.3 Research question The study aimed to answer the following research question: • What is the evidence available for determining clinical utility in the development of assessments in occupational therapy? 1.4 Research aims • To map the evidence available for determining clinical utility in the development of assessments in occupational therapy. • To identify possible gaps within research and the development of assessments. • To use the results of this study to advocate for the importance of determining the clinical utility of assessments. 1.5 Research objectives The following objectives were used to guide the study: • To determine whether clinical utility is included in psychometric testing during assessment development in occupational therapy. • To map the available literature on clinical utility in occupational therapy assessments. 1.6 Significance of the study An assessment with good clinical utility should be used to gain appropriate and accurate results in order to provide a service that aligns with the Health Professions Council of South Africa (HPCSA) good practice guidelines and that is fair and just to the population (Foxcroft, 2012; HPCSA, 2008; Van Jaarsveld et al., 2012). Clinical utility is an emerging field that should be included within the development of assessments to ensure that an assessment is applicable to the population for which it is being used. This is important, as the results of assessments impact the practitioners‘ clinical decisions, such as planning appropriate interventions, monitoring progress and planning discharge (Bowyer et al., 2012; Foxcroft & Roodt, 2013; Kramer et al., 2009). This study will investigate the evidence available for determining clinical utility within occupational therapy assessments. This aims to help identify possible gaps within 4 research and the development of assessments. The results of this study could be used to advocate for the importance and need of determining clinical utility of assessments. This is essential within the diverse context of South Africa and would ultimately assist in ensuring that appropriate, efficient assessments are used for the South African population. 1.7 Outline of this research report The research report consists of six chapters in total. Chapter one provided the introduction. Chapter two will provide an in-depth literature review on the key components of the study. Chapter three will report on the methodology used for this research report, including the search strategy and the data collection process. In Chapter four the results will be illustrated through the use of table and figures to summarise the key findings. In Chapter five, the results will be discussed based on the research objectives of the study. Lastly, Chapter six will provide a conclusion of what the results of this study mean for clinical practice and recommendations will be made for future research. The following diagram indicates the outline of this research report. Figure 1.1 Outline of the research report Chapter 1 Introduction Chapter 2 Literature Review Chapter 3 Methodology Chapter 4 Results Chapter 5 Discussion Chapter 6 Conclusion and Recommendations References Appendices 5 CHAPTER 2 : LITERATURE REVIEW 2.1 Introduction This study aims to report on the current evidence available for determining the clinical utility in the development of assessments in occupational therapy, in order to identify possible gaps and advocate for inclusion of clinical utility studies within research. Chapter two includes a critical review of the literature on the role of assessments in occupational therapy, assessment development and the importance of considering the context in this process. Clinical utility as a concept will be explored, as well as where it fits into the assessment development process. Lastly, this chapter will investigate whether clinical utility has been established within assessment development in therapeutic and rehabilitative healthcare services. 2.2 The role of assessment in occupational therapy 2.2.1 Occupational therapy Occupational therapy aims to improve the engagement and participation in daily occupations of individuals and groups (AOTA, 2020). The key focus in occupational therapy is the interaction between the person, occupation and environment (AOTA, 2020; Wong & Fisher, 2015). The occupational therapy process consists of evaluating the client, his occupational performance, as well as his environment, in order to plan and provide client-centred intervention to address specific outcomes (AOTA, 2020). Figure 2.1 illustrates the domain of occupational therapy. Occupation stands central to the profession and is defined as everyday activities that a person engages in that adds meaning and purpose to their lives (AOTA, 2020; Gallagher et al., 2015). During the evaluation step of the therapeutic process, an occupational therapist would determine a client’s occupational profile in addition to their occupational performance (AOTA, 2020). A client’s occupational profile includes the occupations that they need and want to do, whereas the occupational performance is determined by whether a person can engage, as well as how well they are able to engage in those occupations (AOTA, 2020). 6 Whilst assessing a client to determine strengths and weaknesses, the occupational therapist will assess performance patterns, performance skills, and client factors (AOTA, 2020). Performance patterns include factors that could either hinder or aid occupational engagement including habits, routines, roles, and rituals (AOTA, 2020). Performance skills include the motor, process, and social skills that a person needs to perform an occupation within a social context (AOTA, 2020). Client factors are defined as the factors pertaining to a client which will impact their occupational performance and include their beliefs and values, body functions, and body structures (AOTA, 2020). The context in which the occupation takes place, impacts the nature of the occupation. This could either hinder or aid a person’s occupational performance (Whiteford, 2010). Context includes both environmental and personal factors which impact a client’s engagement in occupations (AOTA, 2020). Thus, it is essential to include the contextual and environmental factors in the assessment process, as an occupation does not occur in isolation ( AOTA, 2020; Whiteford, 2010). South African occupational therapists are required to navigate the impact of diverse contexts; whether working in private facilities with medical funding or providing services in rural communities with limited resources (Jansen van Vuuren et al., 2020). Figure 2.1 The domain of occupational therapy according to the Occupational Therapy Practice Framework: 4rd edition (AOTA, 2020: p.7). Occupations •Activities of daily living (ADLs) •Instrumental activities of daily living (IADLs) •Health management •Rest and sleep •Education •Work •Play •Leisure •Social participation Contexts •Environmental factors •Personal factors Performance patterns •Habits •Routines •Roles •Rituals Performance skills •Motor skills •Process skills •Social interaction skills Client factors •Values, beliefs and spirituality •Body functions •Body structures 7 An occupational therapist thus needs to determine if there are any barriers or facilitators that may impact engagement in occupation. Assessments in occupational therapy are used for this purpose (AOTA, 2020; Brown, 2010; Foxcroft & Roodt, 2013; Kramer et al., 2009; Richardson, 2010). Assessments in occupational therapy can be occupation-centred or assess specific body functions and performance skills (Asaba et al., 2017). Occupational therapists assess and provide services to clients across their lifespan, including newborn babies, school-going children, adults, and the elderly (Jansen van Vuuren et al., 2020). Occupational therapy thus services a large and diverse population group including fields such as paediatrics, physical rehabilitation, mental health, hand therapy, and vocational rehabilitation (AOTA, 2020). Assessments used can overlap between different fields or can be unique to the specific field depending on the client’s need (Alotaibi et al., 2009). 2.2.2 Assessments in occupational therapy Assessments are used to determine barriers and facilitators to occupational engagement and performance (Alotaibi et al., 2009; AOTA, 2020; Brown, 2010; Kramer et al., 2009). An assessment predominantly assesses one or more attributes. This is used to determine a client’s level of functioning at various points in time (Scott et al., 2006; Unsworth, 2000). Assessments are not only used at the start of the therapeutic process to determine therapy goals but also to monitor progress, determine the effectiveness of an intervention, apply for medical aid benefits, and determine discharge from therapy (Alotaibi et al., 2009; Asaba et al., 2017; Benson & Clark, 1982; Scott et al., 2006; Unsworth, 2000). An occupational therapist relies on theory-driven clinical reasoning to determine the most appropriate assessment needed for the situation (Alotaibi et al., 2009; AOTA, 2020). They make use of outcome measurements such as standardised assessments, screening tools, questionnaires, and clinical observations. Alotaibi, Reed and Nadar (2009) conducted an exploratory study to determine the reasons therapists choose specific assessments in occupational therapy. The findings of the study indicated that availability, time efficiency, ease of administration and scoring, as well as standardisation were the main factors considered when selecting an assessment 8 (Alotaibi et al., 2009). The choice of assessment is also influenced by the cost of the assessment (purchasing of materials or scoring) and medical aid expectations for reimbursement (Kramer et al., 2009). When selecting an assessment it is thus essential to select an assessment that is suitable to the client, practice setting, context and population (Darzins et al., 2016; Unsworth, 2000). 2.3 Development of contextually appropriate assessments When considering the appropriateness of assessments, it has been noted that assessments should not only have good psychometric properties (Nalder et al., 2017) but should also be contextually appropriate (Darzins et al., 2016; Lecuona et al., 2016; Unsworth, 2000). Occupational therapists often rely on assessments developed by other professions, however, there has been a steady increase in the development of assessments specifically for the occupational therapy domain (Asaba et al., 2017). 2.3.1 Assessment development Contextually appropriate assessments are described in the literature as assessments that have been developed for a specific population within a specific context (Pascoe & Norman, 2011). If no contextually appropriate assessments are available, internationally developed assessments are often adapted to suit the needs of the therapists (DeVellis, 2017; Pascoe & Norman, 2011). The assessment development process is dynamic and further investigations into the use of an assessment could result in adapting the assessment tool in order to make it more appropriate for a specific setting (Foxcroft, 2012). The limited information and training in assessment development has an impact on the willingness of therapists to develop new assessment tools. Therapists would often rather adapt an existing assessment to suit the situation (DeVellis, 2017). Foxcroft (2012), noted that a risk of this practice is that it may result in the use of assessments that might not be culturally appropriate and useful within the context. When using a standardised assessment, the clinician is expected to conduct the assessment in a standardised manner according to the manual and will compare the client’s results to the norms of the assessment (Pascoe & Norman, 2011). If the assessed client is from a different population or context than the clients from whom the normative data was collected, it could impact on the fairness and usefulness of the information (Evetts et al., 2021; Jorquera-Cabrera et al., 2017; Van Jaarsveld et al., 2012). 9 Switzer et al. (1999) justify the creation of a new assessment if there are no appropriate established measures available or if a nearly appropriate measure cannot be adjusted. When developing a new assessment it is important to note that the assessment tool should provide better outcomes than the existing alternative assessments available (Bossuyt et al., 2012). Benson and Clark (1982) explained that assessment development occurs within four phases, including planning, construction, quantitative evaluation, and validation, with each phase containing multiple steps. Multiple authors have relied on the four-step process to develop and validate assessments (Desrosiers et al., 1993; Kirby et al., 2010; Kruger et al., 2021; Rosenberg et al., 2010; Tungjan et al., 2021; Wren & Benson, 2004). In this proposed model for assessment development, limited emphasis is placed on external or contextual factors that should be incorporated in the development process. In a study done by Tungjan et al. (2021), while using the development process as described by Benson and Clark (1982) they added an additional step to determine the feasibility of the training within a specified population and context. Guidelines such as the Instrument Evaluation Framework (Rudman & Hannah, 1998) as well as the Outcome Measure Rating Form Guidelines (Law, 1987) have been developed to evaluate assessments and ensure a good fit between client, context and assessment. Both the guidelines emphasise the importance of clinical utility as the first phase, which allows the therapist to investigate the clinical utility of an assessment to determine its usefulness (Law, 1987; Rudman & Hannah, 1998). Together, the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education developed a Standards for Educational Psychological Testing that was published in 1999 (Salkind, 2015). This document provides a baseline of criteria that could be used to evaluate assessments. However, there is no legislation that binds assessment developers to abide by the criteria (Salkind, 2015). Evidence-based practice however requires therapists to use assessments that are psychometrically sound (Nalder et al., 2017). As part of the development of an assessment, it is imperative to determine the psychometric properties to ensure that reliable and valid information is gained from the assessment and to make the assessment suitable for the context in which it is used (Switzer et al., 1999) 10 2.3.2 Psychometric testing in assessment development Switzer et al. (1999: p.399) describes psychometric properties as “the properties of the instrument as it functions within the context.” These authors indicate that at a minimum, psychometric requirements when evaluating research instruments or assessments should include reliability and validity (Switzer et al., 1999). The psychometric properties of an assessment tool provide insight into how accurate the assessment tool is and the quality of results that can be expected from the assessment (De Souza et al., 2017; Salmond, 2008). Evaluation of the psychometric properties, mostly including reliability and validity, occur within the different phases of test development and result in continuous refinement (Benson & Clark, 1982). 2.3.2.1 Reliability Reliability refers to the ability of the assessment tool to provide a true score of the constructs being assessed and that the scoring is not altered by the examiner or environmental conditions (Salmond, 2008; Switzer et al., 1999). A reliable assessment tool will allow for accurate interpretation of the results and will allow the examiner to track progress within the same population (Salmond, 2008). Reliability pertains to the stability, equivalence and internal consistency of the assessment (De Souza et al., 2017; Salmond, 2008). The stability of the assessment is determined by test-retest reliability, where it is assumed that the attribute being assessed should remain constant over a period of time (Devon et al., 2007; Salmond, 2008). Test-retest reliability is not relevant where an attribute is expected to change, such as skill improvement following an intervention (Devon et al., 2007). The equivalence of the assessment is determined by inter-rater reliability. Inter-rater reliability is especially important in tests where the examiner is asked to score observations to ensure that different examiners have consistent results (Salmond, 2008). Lastly, if the assessment shows consistency in all items being assessed and the items fit together conceptually, it will have good internal consistency (Devon et al., 2007; Salmond, 2008). Internal consistency can be determined by using Cronbach’s alpha, Kuder Richardson or the Split-half statistical tests (Salmond, 2008). Cronbach’s alpha coefficient is the most commonly used statistical test as it only requires one administration of an assessment (Devon et al., 2007). New assessments should have 11 a coefficient alpha of 0.7 to ensure reliability (Devon et al., 2007). Reliability of an assessment is expressed as Observed Score = True Score +/- Error (DeVellis, 2017; Devon et al., 2007). As no assessment is perfect, the true score can only be assumed (DeVellis, 2017). The method to calculate the error score is dependent on the type of data collected (DeVellis, 2017). The analysis of variance is one method that can be used to determine the error score (DeVellis, 2017). By ensuring good reliability, it will result in an assessment that is predictable and that will provide consistent results (DeVellis, 2017; Devon et al., 2007; Salmond, 2008). 2.3.2.2 Validity The validity of an assessment tool refers to the ability of the tool to assess the construct for which it was developed (Salmond, 2008; Switzer et al., 1999). DeVellis (2017) adds to the definition, by noting that the validity of an assessment can be deduced from how the assessment was constructed, how accurately it can predict specific outcomes, and how it relates to other assessments. There are three types of validity most commonly determined during assessment development include construct validity, content validity, and criterion-related validity (Brown, 2010; DeVellis, 2017; Devon et al., 2007; Salmond, 2008). The various components of validity have, however, been reconceptualised into one concept known as construct validity (Brown, 2010). This has been done to simplify the concept and eliminate the confusion in research between the three different types (Brown, 2010). For the purpose of this literature review, the three types of validity will be explored. However, it is important for occupational therapists to take note of the changes in research terminology. Construct validity determines whether the assessment measures only the construct for which it was developed or whether other constructs are measured as well (Devon et al., 2007; Salmond, 2008). If an assessment measures closely related concepts, the construct validity of the main concept will decrease (Devon et al., 2007). Contrasted groups, testing of hypotheses, factor analysis, and the multitrait-multimethod approach can be used to determine the construct validity of an assessment (Devon et al., 2007). Devon et al. (2007) note the importance of construct validity in assessment development in the field of nursing. Nurses often have to assess concepts that are more abstract in nature and it is essential to ensure that the assessment only measures the intended constructs (Devon et al., 2007). This also relates to occupational therapy 12 assessment development, as this field requires assessment of complex factors to provide holistic intervention (AOTA, 2020). Content validity refers to whether the assessment is measuring all the major components or aspects of the construct (Salmond, 2008; Yusoff, 2019). To ensure that all components are being addressed, the researcher should determine the main components of the construct being measured before developing the assessment (Devon et al., 2007). A panel of experts will review the content of the assessment by scoring each of the items in the assessment (Yusoff, 2019). The researcher will then be able to calculate the content validity index (CVI) and adapt the assessment content accordingly (Devon et al., 2007; Yusoff, 2019). The acceptable CVI scores differ depending on the number of experts used; for two experts the CVI should be at least 0.8, with three to five experts the CVI should be 1 and with nine experts the CVI should be at least 0.78 (Lynn, 1986; Yusoff, 2019). Criterion-related validity refers to whether there is proof of a relationship between the constructs in the assessment and another variable, and if it can predict an outcome (DeVellis, 2017; Devon et al., 2007). Terms such as predictive validity, concurrent validity and postdictive validity are often used synonymously with criterion-related validity (DeVellis, 2017). An assessment shows good criterion-related validity when there is a strong correlation between the predictor and the criterion to support the predicted performance in each criterion (Devon et al., 2007). It is important that criterion-related validity is correctly calculated as criterion contamination leads to poor validity (Devon et al., 2007). The confirmatory factor analysis and the multitrait- multimethod approach can be used to determine the criterion-related validity of an assessment (Devon et al., 2007). The confirmatory factor analysis will result in chi- square value and the multitrait-multimethod approach will provide insight into the convergent and discriminant validity (Devon et al., 2007). Validity is not an all or nothing measurement and each type of validity should be investigated (Devon et al., 2007). Switzer et al., (1999) emphasised that the validity of an assessment tool is dependent on the context in which it is used. It is thus crucial to investigate whether the test is valid and applicable to different groups of people (Salmond, 2008). Bowyer et al. (2012) highlight that a limitation during assessment development is that often only the validity and reliability are included in the 13 psychometric testing and that the assessment developed does not consider the context and investigate the clinical utility of the assessment for healthcare practitioners. 2.3.3 Contextual considerations in assessment development In order to provide client-centred services, the occupational therapist should understand the contextual factors that impact a person’s performance and engagement (Barbara & Whiteford, 2005; Švajger & Piškur, 2016). This is especially important in a multicultural and diverse country such as South Africa (Foxcroft, 2012; Foxcroft & Roodt, 2013; Jansen van Vuuren et al., 2020; Pascoe & Norman, 2011). Occupational therapists in South Africa work in a variety of settings including public and private hospitals, clinics, rehabilitation centres, non-profit organisations, insurance companies, psychiatric institutions, therapy centres, schools, and nursing homes. In each setting, they provide services to clients from different cultural and socio-economic backgrounds (Jansen van Vuuren et al., 2020; Van Stormbroek & Buchanan, 2017;). They have to navigate many contextual considerations, such as poverty, malnutrition and HIV (Jansen van Vuuren et al., 2020; Visser et al., 2016), lower educational levels (Jansen van Vuuren et al., 2020; van der Linde, 2019), as well as minimal funding and resources (Jansen van Vuuren et al., 2020; van Niekerk et al., 2019; Van Stormbroek & Buchanan, 2017). Contextual factors impact the importance of occupations and skills to clients (Bunting, 2016) and need to be included during assessment development. Context includes the social and environmental factors external to the person or assessment (Bunting, 2016; Switzer et al., 1999). This includes the population characteristics for which the assessment is developed, the goal of the assessment, as well as barriers impacting on the administration and scoring of the assessment (Switzer et al., 1999). 2.3.3.1 Population characteristics During assessment development, the population for whom the assessment is developed should be identified and described (DeVellis, 2017). When considering the characteristics of the population, it is important to take into account the age, gender, education level, and life experiences of the prospective clients, as this will impact the administration of the assessment as well as the responses (Evetts et al., 2021; Switzer et al., 1999). Prior to the development of the In Home Occupational Performance 14 Evaluation (I-HOPE), a need for an assessment within the elderly population was noted (Stark et al., 2010). Identifying the population aided the item development phase of assessment development. The authors selected an appropriate method to administer the assessment and included photographic images to be used as visual cues and aid the memory of the older population (Stark et al., 2010). In another article, the language abilities of children with cerebral palsy were noted to have an impact on the way an assessment was administered and that it was more effective to rely on movement observations (Rae et al., 2010). Similarly, Barbara and Whiteford (2005) observed that the language abilities of the population affected the time needed to conduct the Handicap Assessment and Resource Tool in acute hospital settings as it is based on an interview with the client (Barbara & Whiteford, 2005). Van der Linde (2019), ensured that the South African Sensory Integration Screening Instrument (SASISI) included multiple languages for administration as it was specifically developed for low socio- economic communities in South Africa (van der Linde, 2019). It is also essential to consider the cultural appropriateness of an assessment; specifically considering the assumptions, norms, and values of the intended population (Evetts et al., 2021; Switzer et al., 1999). A concern is raised when an assessment was developed using one population and applied to another population (Evetts et al., 2021; He & van de Vijver, 2012). The development of contextually appropriate assessment tools should allow for potential cultural variation and cultural bias (Evetts et al., 2021; Foxcroft & Roodt, 2013). These authors note that culture, however, cannot be separated from the person’s environment and influences how a person thinks and performs tasks (Foxcroft & Roodt, 2013). A culture-free measurement cannot necessarily be developed but the examiners can be sensitive to cultural differences during the assessment development (Foxcroft & Roodt, 2013). Cultural fairness should be strived for to prevent a person from a different cultural background to perform badly in an assessment due to tasks that are unfamiliar to them (Evetts et al., 2021). Assessments are, unfortunately, often developed in the Global North and include minimal cultural variety or diversity in the samples used for normative data (Evetts et al., 2021; Jorquera-Cabrera et al., 2017) . Lastly, when considering the population, historical events and experiences which play a vital role when developing assessments, are often excluded. This impacts the importance of items in an assessment and how the results are interpreted (DeVellis, 15 2017; Switzer et al., 1999). Foxcroft and Roodt (2013), noted that children from low socio-economic backgrounds often scored poorly in psychological measures due to the tasks being unfamiliar and not being seen as important within their circumstances. During the development of the I-HOPE, the authors conducted research to determine the occupations that are most meaningful to the older population (Stark et al., 2010). Munnik (2018) noted that in the process of developing a contextually appropriate school readiness assessment, emphasis had to be placed on the social context as it affected the general development of children. Assessments should include constructs that are meaningful and purposeful for the client (Aplin & Ainsworth, 2018; Munnik, 2018; Stark et al., 2010). Assessments should accommodate for the diverse population and varying socio-economic context within South Africa (Munnik, 2018). 2.3.3.2 Characteristics of the assessment pertaining to administration and scoring The context and resources available within a setting, impact the therapists choice of assessments (Alotaibi et al., 2009). During assessment development, the method of administration and scoring should be adapted to suit the context and the goal of the assessment (Switzer et al., 1999). The cost of the assessment (Bañas & Gorgon, 2014; Chien et al., 2014; Darzins et al., 2016; Ireland & Johnston, 2012; Perlmutter et al., 2013), as well as the time needed to administer the assessment (Atler et al., 2017; Barbara & Whiteford, 2005; Bowyer et al., 2012; Darzins et al., 2016; Ireland & Johnston, 2012; Perlmutter et al., 2013), should be appropriate for the context. If an assessment is too expensive or takes too long to administer in a low socio-economic environment, it will create a barrier to the services rendered (Foxcroft & Roodt, 2013). In the field of paediatric occupational therapy, assessments are often developed in the Global North (Jorquera-Cabrera et al., 2017), resulting in expensive assessments that are not appropriate in the South African context (van der Linde, 2019). The socio-economic status of the proposed population should be assessed and considered in the assessment development process (Foxcroft & Roodt, 2013). The language in which an assessment is administered should be evaluated (Foxcroft & Roodt, 2013). It is noted that assessment results could be altered if an assessment is conducted in a language other than a client’s home language. Even if a client has 16 working knowledge of the language the test is conducted in, it may still affect the processing speed (Foxcroft & Roodt, 2013). Translation has been seen to be appropriate and applicable within certain assessments, however should be used with caution as not all languages have the same concepts or expressions (Foxcroft & Roodt, 2013). Bilingual or multilingual assessments are seen as the best solution, however hold challenges regarding psychometric properties (Foxcroft & Roodt, 2013). It is essential to identify the physical resources and environmental barriers in the context during assessment development. During the development of a contextually appropriate basic Wheelchair Service Provision Test, it was essential to consider the knowledge of physical components that the healthcare providers needed in order to provide effective wheelchair training (Gartz et al., 2016). All aspects of the context, including the socio-economic status, physical resources and language spoken, should be considered and accounted for in assessment development to ensure that a fair and just method of evaluation is used (Carter et al., 2005; Foxcroft & Roodt, 2013; Pascoe & Norman, 2011). The assessment development process should include psychometric testing to ensure that the assessment provides valid and reliable information to aid the therapeutic process (De Souza et al., 2017; Salmond, 2008; Switzer et al., 1999). It is also essential to determine the context in which the assessment will be conducted and ensure that the assessment is applicable to the population, the needs of the setting, as well as the resources available within the setting in order to develop a contextually appropriate assessment (Alotaibi et al., 2009; Foxcroft, 2012; Foxcroft & Roodt, 2013; Switzer et al., 1999). 2.4 Clinical utility Clinimetric properties, is a term that has been noted in the latest research in therapeutic and rehabilitative healthcare services and is used to describe the combination of the psychometric properties, as well as the clinical utility of an assessment (Bañas & Gorgon, 2014; Miller et al., 2014; Romli & Wan Yunus, 2020; Slater et al., 2010;). Clinical practice is moving towards a more evidence-based practice (Nalder et al., 2017), resulting in a larger emphasis on assessments that are not only psychometrically sound but also have good clinical utility (Bossuyt et al., 2012; Glover & Albers, 2007; Nalder et al., 2017). 17 2.4.1 Clinical utility as a concept Clinical utility is a developing concept and has recently been appearing in healthcare research with regards to assessments and interventions (Bowyer et al., 2012; Smart, 2006). Clinical utility of an assessment can be described as how useful and appropriate the assessment is within a specific setting or context (Benson & Clark, 1982; Darzins et al., 2016; Lesko et al., 2010; Macy, 2012; Nalder et al., 2017). Clinical utility impacts the use of assessments within clinical practice (Law, 1987; Tyson & Connell, 2009b), as clinical utility is specific to a context, as well as the stakeholders involved (De Souza et al.,2017; Lesko et al., 2010; Smart, 2006). If an assessment has good clinical utility it will result in good clinical management (Law, 1987), and positive health outcomes for patients (Bossuyt et al., 2012; Nalder et al., 2017) by guiding clinical decision- making and treatment plans (Darzins et al., 2016; Nalder et al., 2017). 2.4.2 Clinical utility in assessment development Assessments are usually developed for a specific context and purpose. The appropriateness of an assessment in a different context cannot be assumed without investigation into the clinical utility (Foxcroft & Roodt, 2013). Bowyer et al. (2012) note that very few studies on clinical utility have been carried out. It is, however, noteworthy that no formal guidelines are available regarding the process of investigating or determining the clinical utility of an assessment (Bowyer et al., 2012; Smart, 2006). Randomised controlled trials are seen as the best method to determine the clinical utility of diagnostic tests, especially in haematology and genetics testing (Peabody et al., 2019). However, in in therapeutic and rehabilitative healthcare services, researchers often conduct qualitative research to determine clinical utility (Glover & Albers, 2007; Nalder et al., 2017; Toomey et al., 1995). Qualitative research in determining clinical utility would focus on: challenges or barriers experienced during administration and scoring, the time needed for the assessment, costs involved and training needed as well as how the results guided their clinical practice (Foxcroft, 2012; Glover & Albers, 2007; Nalder et al., 2017; Toomey et al., 1995). There has also been an increase in meta-data analyses determining the clinical utility of assessments by extracting specific data from the user manuals of the assessments (Bañas & Gorgon, 2014; Bellagamba et al., 2020; de Klerk et al., 2018; Ireland & Johnston, 2012; Jorquera-Cabrera et al., 2017; Kruger et al., 2021). Meta-data analyses are however mostly conducted after an assessment has been developed and researchers want to 18 determine the applicability of the assessment (Bañas & Gorgon, 2014; Bellagamba et al., 2020; Chien et al., 2014). As noted from studies done on clinical utility, there is limited consensus on what the components of clinical utility are, and no set standard of what components to include within research (Barbara & Whiteford, 2005; Bowyer et al., 2012; Corben, Downie & Fielding, 2011; Darzins et al., 2016; Gustafsson et al., 2010; Smart, 2006). However, when determining clinical utility, a judgement is made about the effectiveness, economic impact and work practices needed to conduct the assessment (Smart, 2006; Toomey et al., 1995). Smart (2006) proposed a multidimensional model for investigating clinical utility in healthcare, specifically determining the appropriateness, accessibility, practicability and acceptability of an assessment. Bowyer et al. (2012) suggest that clinical utility should be determined on every level during assessment development and changes made accordingly. Figure 2.2 illustrates the process of how to incorporate establishing clinical utility within each step of the assessment development process. During the initial test development, researchers should take into account the type of data that practitioners require, the method of administration, and the impact that the results will have on clinical practice (Bowyer et al., 2012). The initial assessment should be piloted and feedback from practitioners should be used to bring adaptations where needed (Bowyer et al., 2012). Once the final assessment has been developed, an in- depth analysis of the clinical utility should be done (Bowyer et al., 2012). This will provide insight into how the test is being used in clinical practice and the impact of the results on intervention planning (Bowyer et al., 2012). 19 Figure 2.2 The proposed steps to determine clinical utility alongside the psychometric properties of an assessment tool (Bowyer et al., 2012: p.21). Published with permission from the author. Assessments can be critically evaluated using the Instrument Evaluation Framework (Rudman & Hannah, 1998) or the Outcome Measure Rating Form Guidelines (Law, 1987). The Instrument Evaluation Framework encourages therapists to evaluate the clinical applicability, specificity, availability, time demands, and acceptability to clients, before determining the psychometric properties of an assessment (Rudman & Hannah, 1998). The Outcome Measure Rating Form Guidelines focus on the format and clarity of the assessment, the training needed, the cost of the assessment, as well as the acceptability of the assessment to the clients (appropriateness for the age and developmental level of the clients and the time needed to administer the assessment) (Law, 1987). Since development, these guidelines have not been updated, however set a valuable starting point for the evaluation of assessments in therapeutic and rehabilitative healthcare services. 2.4.3 Clinical utility in therapeutic and rehabilitative healthcare services Therapeutic and rehabilitative healthcare services have shown emerging literature on clinical utility. In the field of physiotherapy, various studies have been conducted to determine the clinical utility of gross motor assessments in children with cerebral palsy (Chrysagis et al., 2014), balance tests (Tyson & Connell, 2009b), as well as walking and mobility tests (Tyson & Connell, 2009a). Toomey et al. (1995) and Glover and Albers (2007) confirm that factors influencing clinical usefulness include the clarity of Steps to ensure psychometric soundness Delineate purpose Specify context Develop data collection format Develop format for quantifying data Examine reliability and validity Steps to ensure clinical usefulness Delineate how assessment will be used in clinical practice Cross-reference with what practitioners need to know about client Determine if data collection procedures are feasible in a given practice context Determine whether format for quantification is relevant and suportive to clinical decisions Examine who is using the assessment in specific contexts and the impact on clinical decisions and best practice 20 administration procedures, the ease of the administration and scoring as well as the time needed for the assessment. A systematic review by Tyson and Connell (2009) investigated the clinical utility of assessment tools used for measuring balance. They placed emphasis on the cost of the assessment, the time needed to complete the assessment, the need for specific training or equipment, and the ease with which the assessment tool can be transported to the client (Tyson & Connell, 2009b,a). Another author also noted the importance of investigating the applicability of the methods of administration, as well as the language in which the test is administered (Foxcroft, 2012). In the field of nursing, the clinical utility of pain scales has been investigated. Lempinen et al. (2020) investigated the clinical utility of the Finnish Version of The Face, Legs, Activity, Cry, Consolability (FLACC) scale in a Paediatric Intensive Care Unit (PICU) in Finland. The study indicated that the pain scale had good clinical utility as the results impacted the clinical decisions made by the nurses (Lempinen et al., 2020). It also noted that the intubation of children did not impact the clinical utility of the scale (Lempinen et al., 2020). The clinical utility of the Pain Assessment Tool (PAT) was determined in a Neonatal Intensive Care Unit (NICU) in Australia in neonates who were ventilated, sedated, and who received muscle-relaxants (Devsam & Kinney, 2021). Surveys and focus groups were used and focused on the administration, scoring, interpretation and impact of results on clinical judgement (Devsam & Kinney, 2021). Clinical utility of pain assessments for people with dementia requires further testing as limited evidence was noted in a scoping review (Zwakhalen et al., 2006). Literature searches indicate that psychology has investigated the clinical utility of the Personality Assessment Inventory (Karlin et al., 2005), the Strengths and Difficulties Questionnaire (Murray et al., 2021) and the Inventory of Depression and Anxiety Symptoms (IDAS) (Stasik-O’Brien et al., 2019); and in social medicine the clinical utility of the Migraine Disability Assessment (MIDAS) (Lipton et al., 2001). Often no specific context or population is provided in these studies, and even though clinical utility is used in the title, no reference is made to clinical utility in the article (Karlin et al., 2005; Lipton et al., 2001; Murray et al., 2021; Stasik-O’Brien et al., 2019). Thus, not providing quality evidence of the clinical utility of assessments in therapeutic and rehabilitative healthcare services. 21 2.5 Clinical utility in occupational therapy In the development of assessments, occupational therapists have become more aware of the value of standardised assessments complementing clinical observations, and have subsequently increased their technical knowledge of assessment development (Asaba et al., 2017; Benson & Clark, 1982;). Ensuring that the assessments have good psychometric properties is included in the development process (Bowyer et al., 2012). However, with the recent shift to include context-appropriate assessments (Foxcroft, 2012), research in exploring the clinical utility of assessments has increased. Assessments are often developed by academics who place emphasis on sound psychometric characteristics, rather than developing an assessment that is useful and applicable within the context for which it has been developed (Bowyer et al., 2012). Having an assessment that is psychometrically sound and has good clinical utility, will be beneficial to therapists’ report writing, justification of services, and motivating for services or equipment (Kramer et al., 2009). In occupational therapy, numerous assessments have been developed specifically for the need of each field as well as assessments that can be used across various fields (Asaba et al., 2017). However, limited research has been conducted in determining the clinical utility of assessments in occupational therapy (Bowyer et al., 2012). In the field of paediatric occupational therapy, sensory integration forms a large part of assessment and treatment (Ayres, 1995). The normative data of sensory integration assessments were collected from children living in the United States of America (USA) (Jorquera-Cabrera et al., 2017; Van Jaarsveld et al., 2012). None of the most commonly used paediatric assessments had data regarding the validity and reliability relevant to the South African context (Jorquera-Cabrera et al., 2017; van der Linde, 2019), and only the Sensory Integration and Praxis Tests (SIPT) have been adapted for use within the South African context (Van Jaarsveld et al., 2012; van der Linde, 2019). The SASISI is a newly developed screening tool that has been developed for use within low socio-economic environments within South Africa (van der Linde, 2019). During assessment development, the clinical utility of the SASISI in public schools was investigated by conducting in-depth interviews with the research assistants (van der Linde, 2019). Recently a study was conducted to determine the clinical utility of the Sensory Profile (SP), the Sensory Processing Measure (SPM) and the Sensory Experiences Questionnaire (SEQ) (Evetts et al., 2021). These sensory modulation 22 measurements were not specifically developed for the South African context, however, they are a good fit due to the costs involved and the time required (Evetts et al., 2021). This study determined the clinical utility of the three assessments for children with autism spectrum disorder in South Africa based on the dimensions of clinical utility as set out by Smart, (2006). An electronic survey was used to determine the perspectives of occupational therapists trained in sensory integration (Evetts et al., 2021). Bowyer et al. (2012) conducted a mixed methods study to determine the clinical usefulness of the Short Child Occupational Profile (SCOPE). The study included occupational therapists from the Global North with 12 years of experience as their expert group; however, the focus groups included other disciplines including physical therapy, social work, speech and language therapy, as well as teachers. The researchers aimed to determine the benefits of using the SCOPE, as well the barriers preventing stakeholders from using the SCOPE (Bowyer et al., 2012). A qualitative research study was conducted to determine the clinical utility of the Canadian Occupational Performance Measure (COPM) in Ottawa-Carleton’s Home Care Programme (Toomey et al., 1995). It was investigated whether the type of assessment (semi-structured interview) was applicable to the setting, whether it was applicable to the population (elderly residents with and without cognitive impairments), and whether the outcomes were of value to the therapist (Toomey et al, 1995). Burton and Tyson (2015) conducted a systematic review to investigate the assessments available for screening cognitive impairments after a stroke. The authors scored each assessment’s clinical utility based on predetermined criteria such as training needed, the time needed to administer, as well as costs involved (Burton & Tyson, 2015). The specific context for which the clinical utility was determined was not mentioned in the study, making the results not as valuable as clinical utility is context-specific. Nalder et al. (2017) conducted a qualitative study with clinicians working in Canada and the United Kingdom (UK) to determine the clinical utility of the Multiple Errands Test (MET). Semi-structured interviews were held with clinicians and topics such as the ease of administration and scoring, as well as challenges experienced, and how the results guided their clinical practice were addressed (Nalder et al., 2017). Investigation into the clinical utility of each assessment should be done when used in 23 a different context than the context for which it was developed, however, limited literature is available on clinical utility in occupational therapy (Bowyer et al., 2012). 2.6 Summary Occupational therapy is a holistic profession that is focused on the transactional relationship between the person, occupation, and environment in order to determine facilitators and barriers to occupational engagement (AOTA, 2020; Brown, 2009; Wong & Fisher, 2015). The initial step of the occupational therapy process is to conduct an evaluation (Alotaibi et al., 2009; AOTA, 2020). Standardised assessments and clinical observations are crucial in guiding intervention, in order to provide effective and just treatment within all fields of occupational therapy (Alotaibi et al., 2009; AOTA, 2020; Lecuona et al., 2017; Van Jaarsveld et al., 2012). An assessment is conducted to determine the areas that require intervention and is based on the domains of occupational therapy, including occupations, contexts, performance patterns, performance skills and client factors (AOTA, 2020). The appropriate assessment is chosen based on the availability, time, and effort required to administer and score the assessment, the cost of the assessment and whether it is standardised; as well as the expectations of the medical aids (Alotaibi et al., 2009). Literature indicates that there has been a recent shift in evaluating diagnostic assessment tools, as the assessments are not only expected to provide accurate information but also useful information relevant to the context in which it is used (Bañas & Gorgon, 2014; Bossuyt et al., 2012; Darzins et al., 2016; de Klerk et al., 2018; Glover & Albers, 2007; Miller et al., 2014; Nalder et al., 2017; Romli & Wan Yunus, 2020; Slater et al., 2010). Occupational therapists often rely on assessments created by other professions (Asaba et al., 2017) and most assessments are developed in the Global North. However, the literature indicates an increase in assessment development within the field of occupational therapy (Alotaibi et al., 2009; Asaba et al., 2017; Benson & Clark, 1982). The increase in newly developed assessments will result in an increase in profession-specific assessments. Occupational therapists rely on the available guidelines to develop assessments (Benson & Clark, 1982; DeVellis, 2017; Devon et al., 2007; Switzer et al., 1999). These guidelines include clear steps on how to conduct psychometric testing, including the testing of the validity and reliability of the assessment (Bowyer et al., 2012; DeVellis, 2017; Devon et al., 2007). These guidelines note that the context should be taken into account (DeVellis, 2017; Switzer 24 et al., 1999), however, no clear guidelines are available for investigating the clinical utility of an assessment tool during the development process (Bowyer et al., 2012; Lesko et al., 2010). Occupational therapists in South Africa mostly rely on assessments that have been developed in the Global North (Jorquera-Cabrera et al., 2017; Pascoe & Norman, 2011). This often results in assessments having to be adapted to suit the needs of the South African population (Foxcroft, 2012; Foxcroft & Roodt, 2013; Pascoe & Norman, 2011). South African therapists work in a variety of settings and provide services to clients who come from different cultural and socio-economic backgrounds (Jansen van Vuuren et al., 2020; Pascoe & Norman, 2011). Very limited research has been conducted to determine the clinical utility of occupational therapy assessments within the South African context. This study will provide a comprehensive summary of the available literature on determining clinical utility in occupational therapy and emphasis possible research gaps in order to advocate for further research. 25 CHAPTER 3 : METHODOLOGY 3.1 Introduction This chapter provides the methodology used in this research project in order to address the following objectives: • To determine whether clinical utility is included in psychometric testing during assessment development in occupational therapy. • To map the available literature on clinical utility in occupational therapy assessments. 3.2 Study design A scoping review was selected for this research study. A scoping review allowed the researcher to examine and map the available research on clinical utility within occupational therapy (Munn et al., 2018; Peters et al., 2020; Tricco et al., 2018). It also allowed the researcher to identify gaps within the literature and advocate for future research studies to aid assessment development within occupational therapy (Munn et al., 2018; Peters et al., 2020; Tricco et al., 2018). Clinical utility is an emerging topic within assessment development and no formal guidelines have been set with regard to investigating the clinical utility of an assessment (Bowyer et al., 2012; Smart, 2006). A scoping review is thus an ideal research design, as it could be used to identify the available research, and determine the methodology used to conduct the research (Munn et al., 2018; Peters et al., 2020). Synthesising the available literature will aid future research and occupational therapy assessment development (Munn et al., 2018; Tricco et al., 2018). Mapping the available data can provide guidelines for investigating clinical utility as it may also provide insight into the methodologies most commonly used in research. It will also provide a summary of the assessments that have been used in clinical utility research. A scoping review entails a detailed search of the literature using various databases, evaluating articles based on predetermined inclusion criteria, and mapping the literature to create an overall view of the available resources and information (Munn et al., 2018). The Joanna Briggs Institute (JBI) Guidelines was used be used to guide the 26 methodology of this research project (Peters et al., 2020), in order to ensure good methodology and transparent reporting of the research project (Tricco et al., 2018). 3.3 Inclusion and exclusion criteria 3.3.1 Inclusion criteria Prior to conducting the scoping review, inclusion criteria pertaining to the population, concept and context was set (Peters et al., 2020). This allowed the literature search to be concise and relevant to the research aim and objectives (Peters et al., 2020). The articles considered for this research study met the following inclusion criteria relating to population, concept and context. 3.3.1.1 Type of evidence sources Published and unpublished (grey literature) literature between January 2005 and December 2020 including clinical utility within occupational therapy was reviewed for the study. A broad time period was selected to include the development of major assessments within occupational therapy. 3.3.1.2 Concept The concept explored in this research study included clinical utility as part of validity testing in assessment development. 3.3.1.3 Context Articles published specifically within the field of occupational therapy were used for the scoping review. Occupational therapy includes various domains such as paediatrics, physical rehabilitation, mental health, vocational rehabilitation and hand therapy, with specific assessments and questionnaires in each domain. All the domains of occupational therapy were included in the scoping review. 3.3.2 Exclusion criteria Articles that were not freely available through the University of Witwatersrand library, as well as articles published in languages other than English, were excluded from the study. 27 3.4 Data collection procedure 3.4.1 Search strategy The three-step search strategy as set out in the Joanna Briggs Reviewer’s Manual was used to perform a comprehensive search of the literature (Peters et al., 2020). This included an initial limited search to determine keywords and search terms, an extensive literature search and lastly reviewing reference lists of included articles for additional articles (Peters et al., 2020). A librarian from the Wits Health Sciences Library was consulted to assist with selecting search terms and conducting a comprehensive search strategy. Throughout the literature search, record of the specific searches as well as changes made were kept in an Excel spreadsheet to ensure rigour in reporting. Only articles published in English were included in the search. The search process was reviewed by the researcher’s supervisor. 3.4.1.1 Step one of data collection Step one of data collection included an initial search to elaborate, and expand the keywords and index terms that were used for the search process. The initial search terms were derived by identifying the main concepts from the research question. Synonyms and words similar in meaning were used to expand the terms in order to have an all-inclusive search. The initial phase was conducted using CINAHL and Pubmed. Search terms were compiled in order to conduct Step two of the data collection. Table 3.1 Search terms for Step two of data collection Key concept Clinical utility Assessment Occupational therapy Search terms Clinical utility Validity testing Ecological validity Assessment Test Instrument Evaluation Occupational therapy/OT 28 3.4.1.2 Step two of data collection The search terms identified in step one of the data collection were combined into a search phrase; “Clinical utility” OR “Validity testing” OR “Ecological Validity” AND assessment* OR test* OR instrument* OR evaluation* AND “occupational therapy” OR “OT”. Boolean operators, truncation, parenthesis and wildcards were used whenever appropriate. A comprehensive search of the following databases was conducted: Pubmed, CINAHL, ClinicalKey, ProQuest, MEDLINE, Cochrane Library and OT seeker. Grey literature from Mednar and Microsoft Academic were included. The full search history is included in Appendix A, however, a breakdown of the articles found per database can be seen in Table 3.2. The comprehensive search yielded a total of 810 articles. After duplicates were removed, article titles and abstracts were screened for relevance. The following inclusion criteria were used: (i) articles pertaining to clinical utility of assessments stated in either the title of the abstract, (ii) articles assessing the clinical utility of assessments (clinical utility of intervention methods were excluded), (iii) articles presenting information about occupational therapy specific assessment (articles related to medicine, pharmacology and other fields of therapeutic and rehabilitative healthcare services were excluded), (iv) meta-data analysis comparing the psychometric and clinimetric properties of assessments, and (v) only articles published between January 2005 and December 2020 were included. Full-text articles were reviewed and non-relevant articles were excluded with reasons. Articles were excluded when the article only included other psychometric testing (n=31) such as validity, reliability, ecological validity, comparison of two tests to name a few. Articles were also excluded if clinical utility was stated in the title, but no investigation of clinical utility was made in the article (n=4). Clinical utility studies of intervention methods, learning programs and approaches were excluded (n=8). Lastly, studies including protocols, brief reports and column articles were excluded as they contained no results (n=3). 29 Table 3.2 Articles found per database Database Total articles Pubmed 162 CINAHL 92 ClinicalKey 56 ProQuest 178 Cochrane Library 12 OT seeker 4 MEDLINE 158 Mednar 71 Microsoft Academics 77 810 3.4.1.3 Step three of data collection The reference lists of the included articles were reviewed for additional articles that met the inclusion criteria and a total of 25 articles were identified. The abstracts of the identified articles were reviewed; however, no full articles could be retrieved. No additional articles were included. 3.4.1.4 Inclusion of grey literature User manuals of assessments, frequently used within the South African context, that were available from the University of the Witwatersrand were reviewed. From the available manuals, 10 user manuals were published within the timeframe of this study. The user manual review yielded one manual that provided insight into the clinical utility of the assessment (Squires et al., 2009), however, no formal clinical utility studies were included in the assessment manuals. 3.4.2 Summary of data collection A scoping review was conducted using the three-step data collection process as set out in the Joanna Briggs Institute Guidelines (Peters et al., 2020). A comprehensive search of published and unpublished literature from nine databases, as well as a review of user manuals available from the occupational therapy department of the University of the Witwatersrand, was carried out. Applicable articles were selected, and the search yielded 37 included studies and one user manual. The overall data collection process, indicating the included and excluded articles, is summarised in the 30 Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flow diagram (Figure 3.1) (Moher et al., 2009). 3.5 Data extraction Data extraction was guided by the aims and objectives of the study. Thus, following the literature search, the included articles were charted in a table: specifically including the author, year of publication, field or domain of occupational therapy, aims or purpose of the study as well as the methodology and sample size of the study (Appendix B). Outcomes related to the study included extracted data of the various components of clinical utility that were addressed by the article. The key findings of the research specifically related to the study objective of whether the clinical utility was determined during the assessment development process or after the assessment had been developed. Data extraction was done by the primary researcher and checked by the supervisor. 3.6 Data analysis and synthesis After extracting and charting the data according to the table, the data was analysed and synthesised by the primary researcher. Descriptive statistics were derived regarding the year of publication, the country where the research was conducted as well as the methodology and sample size used. Descriptive statistics were also used to present the representation of the different domains within occupational therapy, as well as whether the clinical utility study was conducted within assessment development. A qualitative content analysis was conducted using basic coding to categorise the data into overall components of clinical utility identified by the studies (Peters et al., 2020). 31 Figure 3.1 PRISMA flow diagram (Page et al., 2021: p.5) Records identified from: Databases (n = 810) Records removed before screening: Duplicate records removed (n = 206) Records screened by abstract and title (n = 604) Records excluded (n = 488) Reports sought for retrieval (n = 115) Reports not retrieved: no full text available (n = 32) Reports assessed for eligibility (n = 83) Reports excluded: Assess clinical utility of other instrument/intervention (n = 8) Clinical utility in title but not assessed (n = 4) Other psychometric testing (n = 31) No results (n=3) Records identified from: Citation searching (n = 25) User manual review (n=10) Reports assessed for eligibility (n = 28) Reports excluded: Not applicable (n = 18) Other psychometric testing only (n = 9) Studies included in review (n = 37) User manuals included (n=1) Identification of studies via databases and registers Identification of studies via other methods Id e n ti fi c a ti o n S c re e n in g In c lu d e d Reports sought for retrieval (n = 35) Reports not retrieved (n = 7) 32 3.7 Ethical considerations Prior to conducting the scoping review, a protocol was completed and approved by the University of Witwatersrand. The protocol provided the plan for the scoping review and is important in limiting reporting bias and ensuring rigor (Peters et al., 2020). The researcher aimed to limit reporting bias by having clear guidelines for the intended procedures, having predetermined inclusion and exclusion criteria, as well as keeping record of any changes made during the literature search (Peters et al., 2020). A review pair is required during the search stages of a scoping review to ensure reliable inclusion and exclusion of articles and to limit reporting bias (Peters et al., 2020). As this was an individual research report, a review pair was not used, however, the search process was reviewed by the supervisor to compensate for this limitation. The PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) Checklist was used to ensure methodological rigour and transparency in reporting the results of this study. The PRISMA-ScR Checklist is a revised version of the original PRISMA Checklist, developed specifically for scoping reviews (Tricco et al., 2018). It provides the researcher with clear guidance on what to report while doing a scoping review and aligns with the JBI guidelines (Tricco et al., 2018). An ethical clearance waiver was received from the Human Research Ethics Committee (Medical) of the University of Witwatersrand (Ref: W-CBP-211019-02) (Appendix C). 3.8 Conclusion This chapter provided the methodology as guided by the JBI guidelines (Peters et al., 2020) in order to provide reproducible methods and rigorous reporting. The literature search yielded a total of 38 sources, including 37 articles and one relevant user manual. Relevant data from the articles were extracted and the results found using this methodology will be described in the next chapter. 33 CHAPTER 4 : RESULTS 4.1 Introduction This study aimed to map the evidence available for determining clinical utility in the development of assessments in occupational therapy, to identify possible gaps within research and to advocate for the importance of determining clinical utility during assessment development. The first objective of the study was to determine whether clinical utility is included in psychometric testing during assessment development in occupational therapy. The second objective was to map the available literature on clinical utility in occupational therapy. This chapter provides the results of the literature review, that included studies and user manuals that determined and/or investigated the clinical utility of assessments in occupational therapy. 4.2 Distribution of studies The literature search resulted in n=38 articles that addressed the objectives of this study. In order to provide an overview of the included literature, the distribution of the studies will be discussed with regards to the year of publication, as well as the country in which the research was conducted. 4.2.1 Distribution of articles by year of publication Articles published between January 2005 and December 2020 were included in the literature search. Figure 4.1 provides a visual representation of the distribution of the included articles by the year of publication. Of the included articles, nine articles were published between 2005 and 2010, 15 articles between 2011 and 2015 and 14 articles between 2016 and 2020. Figure 4.1 Distribution of articles by year of publication 0 5 10 15 20 2005 - 2010 2011 - 2015 2016 - 2020 To ta l a rt ic le s Time period 34 4.2.2 Distribution of articles by country Included articles reported research from various countries, including twelve articles from Australia, nine articles from the USA, three articles from the UK, two articles from Sweden and one article from Canada, Denmark, Slovenia and Spain, respectively. One article included participants from the UK and USA (Forsyth et al., 2011) and one article included participants from both the UK and Canada (Nalder et al., 2017). Meta- analysis articles were excluded from this section as the systematic and literature reviews were not limited to specific countries. Figure 4.2 Distribution of articles by country 4.3 Research design, methodology, sample size and population of included articles 4.3.1 Research design and methodology During the literature search, all research methodologies were included. This resulted in studies that used qualitative, quantitative and mixed method research designs as well as systematic and literature reviews. Figure 4.3 illustrates the distribution of articles by the methodology used. Of the included articles, 33% used a qualitative approach with either focus groups (Aplin & Ainsworth, 2018; Capdevila et al., 2020; Rodger et al., 2005; Gustafsson et al., 2010; Gustafsson et al., 2012; Rae et al., 2010; Stephans, 2015; Švajger & Piškur, 2016) or interviews (Atler et al., 2017; Barbara & Whiteford, 2005; Forsyth et al., 2011; Nalder et al., 2017). A mixed-method study design was used by 19% of the included articles (Bowyer et al., 2012; Darzins et al., 2016; Grajo, 2015; Hagelskjær et al., 2019; Hamm et al., 2019; Yngve & Ekbladh, 0 2 4 6 8 10 12 14 Australia Canada Denmark Slovenia Spain Sweden UK USA N u m b er o f ar ti cl es Country 35 2015; Zapf et al., 2016) and a quantitative study design with a questionnaire was used by 16% of the included articles (Corben, Downie & Fielding, 2011; Eklund & Gunnarsson, 2008; Gustafsson et al., 2018; Perlmutter et al., 2013; Radia-George et al., 2014; Rowland et al., 2011). Of the included articles, 22% were systematic reviews and 5% were literature reviews. One study did not specify the research methodology (Watkins et al., 2016) and one study used a pre-test/post-test methodology (Doig et al., 2010). Figure 4.3 Distribution of articles by research design 4.3.2 Sample size Scoping and literature reviews were excluded from the sample size calculations. Of the remaining 27 articles and one user manual, three did not specify the sample size used in the research (Doig et al., 2010; Squires et al., 2009; Watkins et al.,2016). The remaining 25 articles were used to provide statistical data on the sample size of the included data. The average sample size per research design is illustrated in Table 4.1. Quantitative 16% Qualitative 33% Mixed-methods 19% Systematic review 22% Literature review 5% Other 5% 36 Table 4.1 Average sample size per research design Research design Qualitative Quantitative Mixed methods Number of articles 12 6 7 Average sample size 10,8 17,3 53,6 4.3.3 Population The included studies determined clinical utility either from the perspective of either the therapist or the client, or relied on information from the assessor’s manual of the assessment. The majority of articles determined the therapist’s perception of the appropriateness of the assessment (Aplin & Ainsworth, 2018; Barbara & Whiteford, 2005;