Measuring the effectiveness of scientific assessments at the knowledge-policy interface

Abstract

‘Scientific assessments’ are elaborate knowledge-policy processes which convene large numbers of experts, policymakers and stakeholders. Their purpose is to surface, deliberate and then organise knowledge relevant to multi-facetted problems of high social importance. Well-known examples include the processes and published reports of the Intergovernmental Panel on Climate Change (IPCC) and the Intergovernmental Science-Policy Platform for Biodiversity and Ecosystem Services (IPBES). Given the widespread and increasingly frequent use of scientific assessments over the past 30 years at global, regional, national and sub-national scales, it is surprising that few empirically based effectiveness evaluations exist. This research asks two questions: Are scientific assessments effective? And, in the South African context, will they help advance our own knowledge-policy interface? To answer the first question, I apply a case-study mixed methods approach to six scientific assessment cases – two at global scale, two at regional scale and two at national scale. This approach integrates quantitative and qualitative data from: 1.) a systematic review (n = 162); 2.) an online survey (n = 674); 3.) semi-structured interviews (n = 43); and 4.) my own experience from involvement in scientific assessments. Using an evaluation framework based on the dimensions of credibility, relevancy, and legitimacy (CRELE) there is overall good evidence supporting the hypothesis that scientific assessments are effective knowledge-policy processes. With high confidence, based on agreement between multiple lines of enquiry, I conclude that specific areas of strength include their ability to engage multi- and interdisciplinary teams in iterative knowledge productionprocedures in a transparent way, producing high-quality scientific outputs which policymakers generally find useful; and which garner a high degree of stakeholder trust. There is slightly less agreement between sources regarding the areas which could be improved. Measured against a hypothetical ‘best-case’ (which may be an unrealistic and unnecessarily high target) specific areas in which there is scope for improvement include the adequate planning of financial and human resources, taking note of ‘assessment fatigue’ and the need for future capacity building; the integration of non-academic knowledge systems or ‘transdisciplinarity’; the provision of more co-production opportunities between experts, policymakers and stakeholders; and the need for innovative public communication and media strategies. With respect to the second question, despite the existence in South Africa of a well-established and resilient knowledge-policy infrastructure, there are nevertheless examples of decision-making processes poorly informed by evidence. These practices primarily subscribe to the ‘test’ model of policy generation, where policymakers rely on legislated minimum requirements rather than adopting more sophisticated, open-ended knowledge production approaches. Given that uneven history, there is little doubt that the targeted application of scientific assessments, for problems that call for a complex response, would substantially help to improve decision-making in South Africa.