Skip navigation

Issues in trial reporting

Reports of randomised controlled trials, and systematic reviews of them, are the way in which we make decisions about whether treatments work, and whether we should use them. Quality issues surrounding trial methods and reports are important - because we are likely to give greater weight to trials of high quality and less weight to those of lower quality.

Ken Schulz and his colleagues have demonstrated that trials which are not randomised exaggerate the estimate of effectiveness by about 40%, and those which are not double-blind exaggerate the estimate of effectiveness by about 17% ( Bandolier 17 ) .

In a new analysis, Schulz and colleagues have examined how well randomised trials of parallel group studies in obstetrics and gynaecology handle issues of blinding, and what happens to patients who are excluded or who 'drop-out' after randomisation [1].

Study

They chose reports from four major obstetrics and gynaecology journals in the 1990 and 1991 issues - 206 of them. Of these 110 were identified for evaluation. Most of the trials were pharmaceutical interventions (72/110).

Blinding

Of the 110 trials, 31 were reported as double-blind, 15 had blinded outcome assessments, one having blinded participants or caregivers, and 63 as not having any form of blinding.

Double-blinding was judged to be feasible in 65 trials, so only about half of the trials that could have been double-blinded were double blinded.

Double-blinding may not always be appropriate or possible. In trials with objective end points (death, for instance), any anticipated gain in blinding may not be worth the additional difficulty or expense. Where outcomes are subjective, or where observers' knowledge of treatment allocation may produce bias, if possible it should be mandatory. Blinding may not be possible - in surgical trials, for instance. Despite this, blinding of assessments is possible and advisable.

Reports which call themselves double blind may not be. An example given is of a report that assigned patients by hospital number and gave two tablets to one group and only one to another. Compromised blinding should always be examined when reading reports.

Protecting treatment allocation

Of the 31 double-blind reports, only eight provided information on the protection of allocation schedule (keeping the randomisation code in a secure place, or not breaking the code until the end of the study). Only five trials reported that blinding had been implemented successfully, and only two tested the efficacy of their blinding - and both found substantial unblinding of the treatment assignments.

Numbers of papers reporting on blinding and exclusions

Number of trials 110
Double blinding possible 65
Double blinding stated 31
Protected treatment allocation 8
Successful blinding stated 5
Blinding tested 2
Excluded at least one patient 52
Insufficient information on exclusion 9
No exclusions 49
Explicit statement that none excluded 11

Exclusions

It is rare for all patients randomised to fulfil all the criteria established at the beginning. These become drop outs, or exclusions. Reports sometimes tell us about what happens to these patients, and then do an analysis of the results on what happened to all patients who fulfilled the trial protocol - a 'per protocol' analysis.

The most rigorous way of dealing with results, the most valid and unbiased way, is to analyse all patients randomised to the originally assigned groups, regardless of compliance with the protocol - the 'intention-to-treat' analysis.

In the Schulz analysis, 52/110 reports had at least one patient excluded. Of these, 29 indicated that more than 10% of patients randomised had been excluded and only 34 said which treatment groups excluded patients had been allocated to. Insufficient information was presented in nine reports to make any judgement. 49/110 trials reported no apparent exclusions. Yet only 11 of these explicitly stated that no exclusions had taken place.

Intention to treat uniquely preserves the ability of randomisation to reduce bias. Bandolier's own experience is that intention to treat has different meanings for different authors, so for safety the rule is to trust most those analyses where all patients randomised are analysed, though per protocol analysis may provide additional useful information.

Guidelines on blinding

Schulz and colleagues suggest that authors should provide the following information on blinding:-
  1. Mechanism - tablets, capsules, tablets.
  2. Similarity of characteristics of treatment - appearance, taste, administration.
  3. Control of allocation schedule - location of allocation schedule during the trial, when the code was broken after the trial, circumstances in which code could be broken for individual patient.
  4. Statement on perceived success or failure of the double blinding efforts.


If these guidelines were followed by authors, it would make the work of the reader and reviewer a great deal easier, and help us get closer to the truth about treatment efficacy and effectiveness.

Reference:

  1. KF Schulz, DA Grimes, DG Altman, RJ Hayes. Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel-group trials in obstetrics and gynaecology. British Medical Journal 1996 312:742-4.



previous or next story in this issue