Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

Evidence-based guidelines?

Variability of guidelines
Evidence-base of guidelines

Bandolier is interested in guidelines and their development. On the face of it they seem like a good idea because they should condense all the best knowledge and experience to give any individual practitioner the confidence that, within limits, they can aspire to the same level of practice as the best in their field. Bandolier's Internet pages have examples of good guidelines [1].

Variability of guidelines

Not all guidelines are created equal and there are examples of great variability between the advice of guidelines. One revelation from Newcastle was that of guidelines for anticoagulation for atrial fibrillation in the UK [2].

In 1996 various people and organisations in England, Wales and Scotland were contacted about the existence of guidelines for anticoagulant treatment of atrial fibrillation. These included regional and national NHS bodies, professional and charitable institutions, and members of mailing lists of audit organisations. They represented purchasers and providers of healthcare and relevant national organisations.

Guidelines were defined as a document produced to help clinicians decide which patients should be given anticoagulant drugs. Drafts, or documents designed for single specialised units, or to provide guidance once warfarin treatment had begun were not included. Where possible, guideline developers were interviewed using a semistructured method about how guidelines had been developed.

All included guidelines were applied to 100 consecutive patients with atrial fibrillation aged 65 years or older identified in a community survey. Details of risk factors for stroke or contraindications for treatment were obtained.


The overall response rate was 66% (350/534), yielding 48 documents of which 20 fulfilled the requirements for definition of a guideline. They varied from a single page to 28 pages, were primarily for use by general practitioners, and affected populations from 12,000 to 500,000.

Guidelines were not systematically developed. A group of people developed about half, and a single person developed the other half. About half had some outside consultation, but about a quarter had no external review. Distribution was haphazard and few had educational meetings to introduce the guideline. Only one was explicitly claimed to be evidence-based, and had outside consultations from a health economist and clinician, with external review and local consultation, with wide distribution and an educational meeting to introduce the guideline.

When applied to 100 consecutive patients, the number recommended for anticoagulation by the guidelines ranged from 13 to 100 (Figure 1). Only one patient would have had anticoagulant treatment recommended by all guidelines, but every patient would have been recommended for anticoagulation by at least two guidelines (but not the same two). Target INR values varied between 1.2 to 1.5 and 2.5 to 3.0.

Figure 1: Anticoagulation guidelines in the UK applied to the same 100 consecutive patients

Evidence-base of guidelines

Another examination of the evidence-base of guidelines comes from Greece [3]. Researchers looked for guidelines published in 1979, 1984, 1989, 1994 and 1999 in six prestigious English language journals (Annals of Internal Medicine, BMJ, JAMA, NEJM, Lancet and Pediatrics). The definition of what constituted a guideline was specific and included all articles containing the words guideline or recommendation or other characteristic words in title or abstract and which had a main focus on prevention or therapeutic interventions.

For each guideline, the reference lists were scrutinised and characterised as randomised trial, systematic review, meta-analysis, or neither. All cited articles were searched in MEDLINE and full records and abstracts were scrutinised. The full paper was retrieved and read in the case of any uncertainty about whether it was a randomised trial. This also applied to articles published before 1966.

Where guidelines had references, contained fewer than two citations of randomised trials, and cited no systematic reviews, a full MEDLINE search was performed up to the year of publication of the guideline.


There were 191 guidelines identified in these six journals for the years searched, predominantly from the USA (86%). Group authorship was most common (84%). Of the 191 guidelines only 12 (6%) had performed a systematic review, but 130 (68%) made no mention about a lack of evidence. Thirty-six guidelines (19%) had no references.

Randomised trials made up a minority of the citations (Table 1). Only 8% of the citations were randomised trials, and fewer than 1% were systematic reviews or meta-analyses of randomised trials or epidemiological studies. The proportion of guidelines not citing any randomised trials fell from 95% in 1979 to 53% in 1999 (Figure 2). Only about one guideline in 10 had a systematic review or meta-analysis.

Table 1: Citations in 191 guidelines in six prestigious medical journals



Percent of total citations

Mean citations per guideline

Total from 191 guidelines 4853 100 25.4
Randomised trial 393 8.1 2.1
Systematic review 19 0.4 0.1
Meta-analysis of randomised trials 23 0.5 0.1
Meta-analysis of epidemiological studies 11 0.2 0.1
Books/pamphlet/brochure 719 14.8 3.8
Abstract 122 2.5 0.6

Figure 2: Guidelines and their citation of RCTs

Thirty-nine guidelines had fewer than two randomised citations with no systematic review. Because 30 were in Pediatrics, 10 of these were chosen at random, and 19 in all were the subject of specific searches. In 12 of the 19, additional relevant randomised trials were found. The number of additional randomised trials was 1 to 194 per topic.


Guidelines are important, and are proliferating. They can be individual, local, regional or national, and often many variants of the same guidelines exist at the same time. Guidelines should be updated regularly. Most importantly, they should be based on the best available evidence. The evidence is that many, perhaps most, are not.

These two papers look at a particular clinical condition exclusively in the UK, and more widely at material that turned out to be predominantly from the USA. They give the same answer, that when examined guidelines are not good enough. The Greek review [2] interestingly looks at parameters associated with citing randomised evidence in guidelines. Those funded by government and professional bodies were worse than those with university and private (usually pharmaceutical) sources of funds. The lesson is that we should not take any guideline on trust, without a sceptical examination of how it has been arrived at, and whether it has followed good practice for guideline development.


  1. R Thomson et al. Decision analysis and guidelines for anticoagulation therapy to prevent stroke in patients with atrial fibrillation. Lancet 2000 355: 956-962.
  2. R Thomson et al. Guidelines on anticoagulant treatment for atrial fibrillation in Great Britain: variation in content and implications for treatment. BMJ 1998 316: 509-513.
  3. IA Giannakakis et al. Citation of randomised evidence in support of guidelines of therapeutic and preventive interventions. Journal of Clinical Epidemiology 2002 55: 545-555.
previous or next story in this issue