Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

Funnel plots: is seeing believing?

Study
Results
Comment

Publication bias is one of those chestnuts that is difficult to deal with. At its most nihilistic the argument is that negative studies are not published, so that any positive finding in a systematic review or meta-analysis will be balanced by many more studies showing no effect: nothing works. Invoking the spectre of publication bias is a useful support for doing nothing.

That is not to say that publication bias does not occur, because it surely does. The problem is knowing what you don't know. One solution has been the use of funnel plots, where some measure of effect of treatment (like odds ratio) is plotted against some measure of size or variance. The claim is that the absence of publication bias produces symmetry, and the presence of publication bias asymmetry.

Expert advice on systematic review and meta-analysis is to check for publication bias using funnel plots, both for writers and readers. Many journals insist on them, despite many publications demonstrating uncertainty and caveats, most of which seem to have gone unheeded. A new study [1] demonstrates the difficulty in recognising asymmetry in funnel plots.

Study

Researchers produced a number of funnel plots based on simulated meta-analyses, each with 10 studies, with sample size generated randomly for individual studies between 50 and 500, with a median of 158. This is typical of many meta-analyses. Publication bias was generated in some by allowing studies with lower P-values and higher sample size to be more likely to be included, and analyses could be statistically homogeneous or heterogeneous.

Twenty-two funnel plots were shown to 41 participants who were undertaking a meta-analysis course, and clinical researchers with some knowledge of meta-analysis. They were asked to grade funnel plots, using written instructions, as to whether there was evidence of publication bias (yes, no, maybe).

Results

About half (53% for all, 55% excluding maybe responses) the plots were correctly identified, with no difference between types of participant. That participants could not identify publication bias from funnel plots was not surprising, because a wide range of asymmetry was present in funnel plots with and without simulated publication bias, or with and without heterogeneity in 1,000 simulated meta-analyses (Table 1).



Table 1: Heterogeneity and publication bias in simulated meta-analyses, and the degree of funnel plot asymmetry



Asymmetry
Heterogeneity
Publication bias
Very high
High
Moderate
Low
No
No
3
13
28
56
Yes
17
18
32
34
Yes
No
4
14
31
51
 
Yes
15
23
33
29


Comment

Basically then, most of us would feel uncomfortable about standing up and giving a quick lecture about funnel plots, and we couldn't spot publication bias in a funnel plot anyway. There is a large literature that confirms the finding in Table 1, that asymmetry exists with and without publication bias, so asymmetry tells us nothing about publication bias. Perhaps we ought to think it out again.

Reference:

  1. N Terrin et al. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology 2005 58: 894-901.

previous or next story