Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

Mindstretcher: analysing observational studies

Study
Results
Comment
How gold is the gold standard?
Implications for interpreting observational studies

Haven't you always wanted to know what goes on in those statistical shenanigans found in so many observational studies, where raw results are 'adjusted' for a whole range of criteria that may affect the results, from age and sex to inside leg measurement. No, it's not sad, but important.

Usually all this is opaque to us, and probably ever will be, but a study of the difference that different methods make, and related to results of randomised trials [1] and an accompanying editorial [2] at least provide an indication of the magnitude of the changes adjusting methods can make.

Study

The study was of 122,000 Medicare patients in the USA aged 65-84, admitted with acute myocardial infarction, and eligible for cardiac catheterisation. Follow up for over seven years allowed investigation of the association between long-term survival and having cardiac catheterisation.

There was a rich set of data on each patient, and this allowed a number of different methods of adjusting raw results for various characteristics to be made.

Results

Of the 122,000 patients, 73,000 received cardiac catheterisation within 30 days. These tended to be younger, to be men, with less severe infarctions, and were more likely to be admitted to high-volume hospitals.

Table 1 shows the results. Without adjusting for differences between those given cardiac catheterisation and those not, the intervention looks very useful, with a very low relative mortality rate. Adjusting for the different risk factors reduced the apparent benefit, with a 64% apparent reduction in mortality falling to approximately 45% reduction in mortality with cardiac catheterisation for most methods, though only a 16% reduction with one method (Table 1). The method of adjusting the crude results obviously had a major impact on the apparent benefits of the intervention.



Table 1: Relative mortality rate (cardiac catheterisation vs no catheterisation) in a large observational study with unadjusted results and results adjusted to take account of imbalances in the two populations



Risk adjustment mothod
Relative mortality rate
Unadjusted survival model
0.36
Multivariate survival model with 65 covariates
0.51
Survival model using propensity score
0.54
Survival model using complex propensity score
0.54
Survival model using propensity based matching cohort
0.54
Instrumental variable-adjusted method
0.84


Comment

The original paper and accompanying editorial [2] are more of a mindbreaker than a mindstretcher for ordinary mortals. The survival benefits of routine invasive care from randomised trials are between 8% and 21%, and the question here is whether these RCT results can be used as the gold-standard to judge the methods of adjusting results from the large observational study.

If so, clearly the instrumental variable method would seem to be best. If not, the implication is that we are all at sea on this one. The argument would be that as randomisation balances all the differences between patients, that balance allows us to see the true effect of the intervention. Yes, but; the buts might include the fact that randomised trials may not reflect every aspect of clinical practice, and that exclusions from trials might muddy the waters when interpreting it for clinical practice.

How gold is the gold standard?

For this example, the effect sizes used from two meta-analyses (8% and 21% benefit) as a gold standard may not be the right ones. For a start, one analysis was over about six months and the other 17 months - much shorter than the seven years (84 months) used in the observational study. Then there is the point that neither meta-analysis exactly mirrors what is going on in clinical practice. A third point is that it is possible to find benefit effects of about 50% in at least one outcome of the meta-analyses. A fourth point, and probably very important, is that the statistical benefits were derived from relatively small absolute benefits.

Implications for interpreting observational studies

Confused? Join the club. The bottom line is that adjusting results in observational studies can make a huge difference to apparent treatment effect, and that it isn't an exact science. A subsidiary is that choosing a gold standard has to be done with care.

In this case, where even the most conservative method indicated a benefit, we can be relatively sanguine. Where observational studies show only a small benefit, or bare statistical significance, we need to be much more cautious about interpretation, and worry more about issues like confounding by indication, or unknown differences between groups that are not taken into account.

References:

  1. TA Stukel et al. Analysis of observational studies in the presence of treatment selection bias. JAMA 2007 297: 278-285. 2 RB D'Agostino, RB D'Agostino. Estimating treatment effects using observational data. JAMA 297: 314-316.

previous or next story