Effectiveness of Training Programmes for and Assessment of Quality of Care Provided by Community Health Workers Undertaking Comprehensive Care in Low- and Middle-Income Countries

Abstract

Background The use of community healthcare workers (CHWs) remains a strategy for improving access to healthcare of vulnerable groups particularly in low- and middle-income countries (LMICs). CHWs are often unlicensed, have minimal training in healthcare, and provide community-based services. To maximise their potential, effective and continuous training is essential. However, the range of CHWs activities is increasing towards more comprehensive healthcare services in many countries. These national comprehensive CHW programmes need standards of practice, training and assessment to ensure quality-of-care. The training programmes designed for CHWs within such national programmes, methods of assessing the training programmes, and how training influences the quality of comprehensive care provided by CHWs, need evaluation. This PhD study thus asks: What are the LMIC experiences of assessing CHW training programmes to enable CHW to provide comprehensive care, and how can the resulting quality-of-care provided by trained CHWs within a comprehensive care programme be assessed? Objectives The PhD study had three parts and aims. 1) To conduct two related systematic reviews: To identify existing LMIC CHW training programmes for comprehensive care and synthesise evidence on their effectiveness and to synthesise evidence on the tools used to evaluate experience and outcomes of CHW training and the utility of such tools. 2) To assess the utility (i.e. validity and reliability) of one recently designed tool for assessing the quality of CHWs comprehensive care in South Africa. 3) To determine messages/activities scores derived from the proportion of health messages given in the households by CHWs who provide comprehensive care in South Africa, and the associated factors. Methods To identify and synthesise evidence on the effectiveness of CHWs training programmes for comprehensive care and assessment methods in LMIC, two related systematic reviews were conducted using a similar search strategy but slightly different screening criteria. The study design applied in the two related systematic reviews included the pre- and post-intervention xi study design, prospective observational, randomised controlled trial, and repeat cross-sectional study design. The search strategy was guided by the PICOS (Population, Intervention, Comparison, Outcome and Study design) framework. The search was conducted in three databases (PubMed, CINAHL and Cochrane Review) from 12 September 1978 (Alma Ata) until end 2023. The Kirkpatrick’s four hierarchies (Kirkpatrick and Kirkpatrick 2006) was used to provide evidence on the utility of post-training assessment tools. The empirical works in this PhD study drew on data from a previous intervention study (Batlhokomedi Project) that was conducted over a 3-year period which included the development of a tool (quality-of-care assessment tool) for assessing the quality of comprehensive services provided by CHWs. To assess the utility of the developed tool, this PhD utilised the repeated measure data collected in the previous project by direct observation of CHWs during 18 household visits. Multilevel modelling and Bland Altman plot were performed to estimate inter- rater reliability (IRR) between paired observers' assigned quality-of-care messages and communication scores. Using primary rating data from 21 CHWs and three team leaders in two primary health care facilities per province I determined the agreement index (should be ≥85%, else, revise the tool), including whether the tool covered everything that happens during their household visits (face validity) and whether they were happy to be assessed using the tool (acceptability). For content validity, using a panel of six experts’ rating data on 11 items of the tool, the content validity index (CVI) of individual items (I-CVI) or entire scale (S-CVI) was calculated. Then, to determine factors associated with the quality of comprehensive care based on data collected using the validated tool, a measure (messages/activities scores) was derived from the proportion of health messages given or activities provided by 34 randomly-selected CHWs during 376 household visits in three South African districts, using a cross-sectional study design. Multilevel models were fitted to identify associated factors of messages/activities score, adjusted for the clustering of observations within CHWs. The models were adjusted for fieldworkers and study facilities as fixed effects. Results On the effectiveness of CHW training, the review included 14 articles from 12 LMICs. Most training packages were on maternal and child health (MCH)-related programmes, now referred to as narrower comprehensive care package. Three studies evaluated trainings on care packages xii encompassing health conditions in addition to MCH (now referred to as wider comprehensive care package). From pre-training assessment, scores improved significantly at post-training (from 5.7 to 62.8 percentage points [pp]). Compared to immediate post training, scores decreased significantly 2-12 months after training (from 12.5 and 34.0 pp). Between assessment periods, scores decreased by 10.3 pp for CHW groups with lower (primary) educational background but increased by 1.8 pp for those with higher (secondary) education. Health professional-led training resulted in 10.3 pp increased score while peer CHW-led training showed 1.9 pp decrease. On the assessment tools and their utility for evaluating experience and outcomes of CHW training, the review included 30 studies from 15 LMICs that have reported any type of tools; however, less than half (n=13, 43.3%) provided any evidence of utility. Majority of the studies were conducted in the Sustainable Development Goals era (2016-2023; n=15, 50%), reported assessment tools on the narrower range of comprehensive care packages (involving maternal and child health-related programmes; n=24, 80%) and across a spectrum of training interventions (initial and/or refresher and/or supportive supervision; 14, 47%). The most commonly used tools were questionnaire-based (including written, self-completed, interviewer administered or tablet- based; n=19) followed by checklists/observation instruments (n=8). IRR (n=9, 69%) was mostly assessed, either singly (n=4) or in combination (e.g., along with face, content validity; n=5). Only one study reported more than two utility types (face/content/construct/IRR/re- test/reproducibility). Of the 13 studies, most (n=12, 92%) used objective measures (i.e., not self- reported) but only one utilised an evaluation model by reporting the Kirkpatrick level of evaluation. Collectively, the findings show that CHWs programmes in LMICs are performing comprehensive care that can be considered under ‘narrower’ and ‘wider’ scopes, and for these, there is evidence of effectiveness of CHW training programmes mostly on the ‘narrow’ comprehensive care scope related to maternal and child health. For these programmes, tools have been developed to assess the gains of training, mostly structured questionnaires and checklists that are self or interviewer-administered or applied while observing the performance of the CHWs during household visits. However, evidence on the utility of these assessment tools is limited, with no validated tool showing evidence of high-quality utility. Regarding the wider comprehensive care scope, limited evidence of comparable level of effectiveness exists but the xiii relevance of well-trained and empowered supportive supervision may be more relevant on an ongoing basis. Availability of evidence of use of evaluation models in comprehensive care training programme is limited, based only on the Kirkpatrick’s four levels of training evaluation. The quality-of-care assessment tool validated within this study has a high acceptability (agreement index ranged from 88% to 100% in three sites but was 64% in one site), face (agreement index ranged from 88% to 100% in all four sites) and content validity (I-CVI was 95%), and the IRR was substantial for quality-of-care messages (77%) but not for communication score (14%). To determine factors associated with the quality of comprehensive care, data collected using the validated quality-of-care messages component of the tool demonstrated important factors related to the messages provided by, and activities of CHWs, across CHW teams. The messages/activities score increased with each extra 5-minute increase in visit duration (1.5%; 95% confidence interval [CI], 0.5 to 2.5, p=0.003) and marginally for having a household member with HIV (4.8%, p=0.067). However, the messages/activities scores decreased in households with either hypertension (5.1%; 95% CI, 0.2 to 10.0, p<0.040), children/babies (10.1%; 95% CI, 4.0 to 16.2, p=0.001), diabetes (15.5%; 95% CI, 9.9 to 21.2, p<0.001), increasing household size (3.6%; 95% CI, 1.6 to 5.7, p=0.001), increasing number of household conditions (21.7%; 95% CI, 18.2 to 25.1, p<0.001) and members with either TB (13.4%; 95% CI, 3.5 to 23.3, p=0.008) or cough (17.5%; 95% CI, 9.7 to 25.3, p<0.001). Conclusions In the current context of task shifting to CHWs, this category of workers with limited training is taking on a much broader remix of work than they were used to including narrower and wider comprehensive care scopes. This study identifies methods and tools, and their utility, to measure effectiveness of training in LMICs, including a tool previously developed but which was validated in this study. The messages component of this tool facilitated the identification of variations in performance of individual CHWs and CHW teams who provide comprehensive care. The tool may be used in summative/formative assessments of CHWs. Such assessments, reflections and discussions with CHW on performance may encourage and support quality of comprehensive healthcare provision. Future studies should explore possible digitisation of the validated tool to facilitate immediate feedback and support of CHWs, peers and supervisors.