Reporting of Statistical Results in the Field of Pain Research

Abstract

Pain research aims to understand the mechanisms, treatments, and impact of pain on individuals and populations. Experimental studies, where researchers manipulate variables and measure outcomes, are crucial in this field. Statistical tests are used to interpret the data and minimize false claims about the effects of manipulations, and the primary output of these tests that is used to minimise such claims in the probability (P) value. Technically, a P-value represents the probability of obtaining the observed data, or more extreme data, assuming the null hypothesis (no effect) is true. Because of this definition, the P-value is frequently interpreted as the strength of evidence against the null hypothesis. A smaller P-value indicates stronger evidence against the null hypothesis. Null hypothesis significant testing uses a P-value cutoff, typically 0.05, to decide if results are statistically significant. However, reliance on this cutoff can lead to misinterpretations about the importance of results. A low P-value is not a guarantee against false positives. Indeed, the threshold creates the problem of false positives. Nor does a small P-value indicate the biological or clinical importance of a result. In my research, I aimed to assess the quality and integrity of statistical reporting in the journal Pain by reviewing P-value reporting in original experimental research articles from 2009-2010 and from 2019-2020 (to account for possible changes with time). My investigation focused on the distribution of P-values according to the type of experimental research (e.g., animal, human), the type of p-value (exact or approximate), and the period when articles were published. I also examined the reporting of test statistics and sample sizes in relation to the type of p-value being reported. Finally, I analysed the distribution of exact P-values and the presence of possible P-hacking (a marker of questionable research practices) using P-curves. My goal was to provide insights into the statistical reporting practices in a leading pain journal. iv All manuscripts published in the journal Pain from 1 January 2009 to 31 December 2010, and 1 January 2019 to 31 December 2020, were reviewed. The articles underwent a screening process based on predefined inclusion and exclusion criteria. Original experimental research papers with at least one identifiable primary hypothesis/objective, and at least one P-value associated with that hypothesis/objective, were included in our study sample. A standardized form developed using REDCap was used for capturing article details, including author names, article title, publication year, volume, issue, page numbers, DOI, study type, objectives, and statistical information. The articles that met our selection criterion were classified into the following categories: basic science (human), basic science (animal), basic science (other), and clinical science. We assessed the quality of P-value reporting and the evidence for P-hacking using the P-curve. Out of 1,342 articles in Pain, 738 met the selection criteria. Of these articles, 98% of experimental papers from 2009-2010 and 97% from 2019-2020 clearly stated a primary hypothesis and had a P-value associated with that hypothesis. Analysis of P-values over time revealed a shift towards reporting exact P-values, with the reporting increasing from 33% in 2009-2010 to 42% in 2019-2020. There also was a positive relationship between the use of exact P-values, instead of approximate P-values, and the reporting of sample size and test statistics with the P-value. The distribution of significant (P < 0.05) and non-significant P- values showed no significant differences across article types. However, a higher proportion of "highly" significant P-values (P < 0.001) was observed in 2019-2020 compared to in 2009- 2010, suggesting either improved research rigor or more selective reporting of significant data. P-curve analysis indicated potential P-hacking in 2009-2010 but not in 2019-2020, highlighting possible changes in research practices over the decade. The findings of this study have important implications for future research in pain management. I assessed data for the leading journal (by impact factor ranking) in the field of pain. The v improvement in the statistical reporting, as shown by the general leaning towards more precise statistical reporting and the absence of P-hacking or publication bias in 2019-2020, as compared to 2009-2010, enhances the credibility of the research in this field. Whether such credibility can be applied across other journals still needs to be established.