Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

How systematic reviews can disappoint

Urge urinary incontinence
Chronic fatigue syndrome
Do decision aids help decision making?

Bandolier tries to find examples of systematic reviews where there is a solid take-home message. After all, we are familiar with uncertainty, and systematic reviews that merely bleat, often inadequately, about inadequacies of research are rather depressing. The trouble is that people use phrases like 'evidence-based medicine', or 'meta-analysis' or 'systematic review' as some form of talisman. Attach one of these phrases to a point of view and an argument is won!

That fails to take into account the fact that reviews can be awful, and even completely wrong. Often systematic reviews try to cover too much ground, and end up giving us too superficial a view of a problem. And even the best of reviews of good studies can leave us in the lurch when there are not enough good studies.

In consequence, here are some examples of systematic reviews that don't help for one reason or another, and on important topics where an answer would be most helpful.

Urge urinary incontinence

A big problem this, and one where we'd like some answers, please. Finding a meta-analysis [1] might make the morning brighter, but reading it made the afternoon stormy.

The review did all the right searching for studies comparing tolteridone versus oxybutinin in the treatment of urge urinary incontinence. There's a clue to a problem, because the review didn't set out to tell us that the treatments worked or how well they worked, but only if one was better than another. Fuzzy thinking from the start.

There were four trials (but two were abstracts). How many patients in these four trials? We have no idea, because we are not told. There may have been some differences favouring oxybutinin for incontinent episodes or tolterodine for voided volume per micturition, and some adverse event data in favour of tolterodine, but without baseline data and numbers it is completely uninterpretable.

This might be the worst meta-analysis ever from a reader point of view, but it badges itself as having been done using 'Cochrane methodology', the ultimate designer label for meta-analysis.

Chronic fatigue syndrome

A review of interventions for chronic fatigue syndrome [2] from York and San Antonio is excellent in many ways. It is beautifully done, with great methods, includes 44 controlled trials (some randomised) and tells us a fair bit about the trials. The problem here is that the 44 studies covered an enormous range of interventions (exercise, drugs, supplements, complementary) plus a wide range of outcomes (psychological, physical, quality of life, physiological and laboratory). If there's a statistical difference, the study gets a tick.

To be fair to the authors of the review, what they have done is what most folk would do. But is it enough? For instance, most studies were small (average about 60 patients, median many fewer). If statistical significance was set at 1 in 20 (5%, 0.05), then that is no great shakes. What would it have looked like at 1%, or 0.01? And what is a useful outcome for chronic fatigue, anyway?

So where's the beef? Let's face it, a significant result in a randomised trial of massage therapy in five (yes, 5, or V) patients doesn't make us sit up and beg for more, especially when we've no idea what benefit means. For what it's worth, behavioural intervention trials were of higher validity and were more often positive.

Do decision aids help decision making?

This review [3] focused on studies in which real patients made actual decisions. The review methods were terrific, the description of observational studies and randomised controlled trials excellent and the studies themselves varied but were often large and asked the important question. Do decision aids influence the decisions that patients take?

There were about 20 studies. The bottom line was that patient decision aids do not influence the decisions that patients take.

But there's more than just this stark answer, and for people involved with patient decisions or interested in using decision aids for patients, this is a wonderful resource. It enlightens about research that has been done, and thinks about research that might be done.


Three systematic reviews and meta-analyses, all on important topics. A mix of quality. What is important is not that they are systematic reviews, but how good they are, and how well they inform. Systematic reviews need brains as well as stamina.

People doing systematic reviews to get another paper on their curriculum vitae or some academic brownie points can be content with useless reviews. If the reviews are about clinical impact rather than about impact factors, then they need to be useful. Can they be used in Pontycymer on a wet Thursday afternoon? That's the ultimate test of quality.


1 MA Harvey et al. Tolteridone versus oxybutinin in the treatment of urge urinary incontinence: a meta-analysis. American Journal of Obstetrics and Gynecology 2001 185: 56-61.
2 P Whiting et al. Interventions for the treatment and management of chronic fatigue syndrome. JAMA 2001 286: 1360-1368.
3 C Eastbrooks et al. Decision aids: are they worth it? A systematic review. Journal of Health Services Research & Policy 2001 6: 170-182.
previous story in this issue