Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

Quality control in systematic reviews

Results
Comment

Bandolier 138 reported a Cochrane disaster, where a review about electronic information for patients reached completely the wrong answer through mistakes. The internal quality control systems of peer review in Cochrane failed to spot it. Well mistakes happen, sometimes large, but more often small. It is good to know that systematic quality control happens in Cochrane, demonstrating no major problems [1].

This study retrospectively repeated the data extraction in all systematic reviews published from the Cochrane Cystic Fibrosis and Genetic Disorders Group. The extraction was done by a statistician experienced in systematic reviews.

Results

There were 42 reviews, of which eight had found no studies to include, so 34 reviews formed the basis of the study. Fifty errors were found in 20 reviews, with 21 different error types. The most common errors were inconsistency between data calculation and results reported in text, and inconsistent reporting of significant versus nonsignificant results. While the errors made for changes to summary results, none altered the conclusions.

Comment

First of all, three cheers for the Cochrane Cystic Fibrosis and Genetic Disorders Group for having the wherewithal to do this quality control exercise, no mean feat. It demanded lots of hard work, and a certain amount of courage. Errors occurred commonly, but made little difference. Peer review by external referees did not spot them, despite four reviewers for each review (consumer, methodologist, statistician, subject expert). Processes have been changed to respond to these findings.

Errors are not limited to Cochrane reviews, or just to reviews. They are common, and perhaps as many as 1 in 10 papers has at least one major error. Minor errors are legion. Readers have to act as their own quality control, and not accept all written words as absolutely and unquestionably true. Question more.

Reference:

  1. AP Jones et al. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology 2005 58: 741-742.

previous story