Winning the Lottery 


' There is much luck in the world, but it is luck. We are none of us safe '.
So said EM Forster nearly 100 years ago.
Bandolier
is constantly astonished in its travels by people who appreciate the importance of
chance, in, say, winning a lottery or avoiding a car accident, but not in clinical
trials. Perhaps it is all down to the way statistics are taught. We should forget
probabilities and pvalues, and acquaint ourselves with more relevant information,
notably how much data do we need to be sure that an observation is not likely to
occur just by chance.
Why are people impressed with pvalues? The cherished value of 0.05 merely says that a result is not more likely to have occurred by chance than 1 time in 20. Most of us have played Monopoly or other games involved with throwing dice. We will have experienced that throwing two sixes with two dice happens relatively often, yet the chance of that is about 1 time in 36.
Look at it another way. If you were about to cross a bridge, and were told that there was a 1 in 20 chance of it falling down when you were on it, would you take the chance? What about 1 in 100, or 1 in 1000? That pvalue of 0.05 also tells you that 1 time in 20 the bridge will fall down.
The dice analogy is pertinent, because there are now (at least) two papers that look at random chance and clinical trials, reminding us how often and how much chance can affect results. An older study actually used dice to mimic clinical trials in stroke prevention [1], while a more recent study [2] used computer simulations of cancer therapy.
DICE 1
In this study [1] participants in a practical class on statistics at a stroke course were given dice and asked to roll them a specified number of times to represent the treatment group of a randomised trial. If six was thrown, this was recorded as a death, with any other number a survival. The procedure was repeated for a control group of similar size. Group size ranged from 5 to 100 patients.The paper gives the results of all 44 trials for 2,256 'patients'. While the paper does many clever things, it is perhaps more instructive to look at the results of the 44 trials. Since each arm of the trial looks for the throwing of one out of six possibilities for standard dice, we might expect that the rate of events was 16.7% (100/6) in each, with an odds ratio or relative risk of 1.
Figure 1 shows a L'Abbé plot of the 44 trials. The expected result is a grouping in the bottom left, on the line of equality at about 17%. Actually, it is a bit more dispersed than that, with some trials far from the line of equality.
Figure 1: L'Abbé plot of DICE 1 trials

The odds ratios for individual trials are shown in Figure 2. Two trials (20 and 40 in total) had odds ratios statistically different from 1. That's one time in every 22 trials, what we expect by chance. Figure 2: Odds ratios for individual DICE studies, by number in 'trial' 

The variability in individual trial arms is shown in Figure 3, where the results are shown for all 88 trial arms. The vertical line shows the overall result (16.7%). Larger samples come close to this, but small samples show values as low as zero, and as high as 60%. Figure 3: Percentage of events in each trial arm of DICE 'trials' 

The overall result, pooling data from all 44 trials, showed that events occurred in 16.0% of treatments and 17.6% of controls (overall mean 16.7%). The relative risk was 0.8 (0.5 to 1.1) and the NNT was 63, with a 95% confidence interval than went from one benefit for every 21 treatments to one harm for every 67 treatments (Table 1). Table 1: Metaanalysis of DICE trials, with sensitivity analysis by size of trial 
Number of 
Outcome (%) with 

Trials 
Patients 
Treatment 
Control 
Relative risk

NNT


All trials  44  2256  16.0  17.6  0.8 (0.5 to 1.1)  62 (21 to 67) 
Larger trials (>40 per group)  11  1190  19.5  17.8  1.1 (0.9 to 1.4)  60 (36 to 16) 
Smaller trials (<40 per group)  33  1066  12.0  17.3  0.7 (0.53 to 0.94)  19 (11 to 98) 

Percent events with treatment 


40 
50 
60 
NNT 
4.2 
2.9 
2.3 
Group size 

25  26  37  57 
50  28  51  73 
100  38  61  88 
200  55  81  96 
300  63  89  99 
400  71  93  99 
500  74  95  100 
With control the event rate was 16%  
At least  
50%  within +/ 0.5  
80%  within +/ 0.5  
95%  within +/ 0.5 
next story in this issue