Skip navigation

Reviewing a Review: DIY tests of Quality

The purpose is to evaluate the scientific quality (i.e., adherence to scientific principles) of research overviews (review articles) published in the medical literature. It is not intended to measure literary quality, importance, relevance, originality, or other attributes of overviews.

The index is for assessing overviews of primary ("original") research on pragmatic questions regarding causation, diagnosis, prognosis, therapy, or prevention. A research overview is a survey of research. The same principles that apply to epidemiological surveys apply to overviews; a question must be clearly specified, a target population identified and accessed, appropriate information obtained from that population in an unbiased fashion, and conclusions derived, sometimes with the help of formal statistical analysis, as is done in "meta-analyses". The fundamental difference between overviews and epidemiological surveys is the unit of analysis, not scientific issues that the questions in this index address.
Quality features
1 Were the search methods used to find evidence on the primary question(s) stated? No Partially Yes
2 Was the search for evidence reasonably comprehensive? No Can't tell Yes
3 Were the criteria used for deciding which studies to include in the overview reported? No Partially Yes
4 Was bias in the selection of studies avoided? No Can't tell Yes
5 Were the criteria used for assessing the validity of the included studies reported? No Partially Yes
6 Was the validity of all the studies referred to in the text assessed using appropriate criteria? No Can't tell Yes
7 Were the methods used to combine the findings of the relevant studies (to reach a conclusion) reported? No Partially Yes
8 Were the findings of the relevant studies combined appropriately relative to the primary question of the overview? No Can't tell Yes
9 Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview? No Partially Yes
10 How would you rate the scientific quality of this overview?



Since most published overviews do not include a methods section it is difficult to answer some of the questions in the index. Base your answers, as much as possible, on information provided in the overview. If the methods that were used are reported incompletely relative to a specific item, score that item as "partially". Similarly, if there is no information provided regarding what was done relative to a particular question, score it as "can't tell", unless there is information in the overview to suggest either that the criterion was or was not met.

For Question 8, if no attempt has been made to combine findings, and no statement is made regarding the inappropriateness of combining findings, check "no". If a summary (general) estimate is given anywhere in the abstract, the discussion, or the summary section of the paper, and it is not reported how that estimate was derived, mark "no" even if there is a statement regarding the limitations of combining the findings of the studies reviewed. If in doubt mark "can't tell".

For an overview to be scored as "yes" on Question 9, data (not just citations) must be reported that support the main conclusions regarding the primary question(s) that the overview addresses.

The score for Question 10, the overall scientific quality, should be based on your answers to the first nine questions. The following guidelines can be used to assist with deriving a summary score: If the "can't tell" option is used one or more times on the preceding questions, a review is likely to have minor flaws at best and it is difficult to rule our major flaws (i.e., a score of 4 or lower). If the "no" option is used on Questions 2, 4, 6 or 8, the review is likely to have major flaws (i.e., a score of 3 or less, depending on the number and degree of the flaws).

Flaws
Extensive Minor  
Major Minimal
1 2 3 4 5 6 7

Reference:

  1. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. Journal of Clinical Epidemiology 1991 44: 1271-8.



previous or next story in this issue