Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

Interpreting pharmacoeconomics


A Bandolier reader wondered why it had not examined a study of pharmacoeconomic submissions to an Australian variety of NICE [1]. The simple answer was that it had failed to swim into our ken, despite our vigilance. So a few years on an opportunity to put right our oversight. What it does is review submissions over the period 1994 to 1997, and highlight errors that were found at quite a high rate.

To some extent the information is out of date, because it partly predated new submission rules. Since 1997 submission of pharmacoeconomic data has become common in many countries (including the UK), and there is better understanding of evidence rules. But there are lessons to be learned, especially by those who have to examine economic analyses and make decisions based on them.


The study was based on 326 major applications to the Australian Pharmaceutical Benefits Scheme between 1994 and 1997. Pharmaceutical companies submit information which is used to determine reimbursement issues. The applications were reviewed in detail, including checking literature, and rerunning searches, validating key assumptions and checking computer or other models. A technical subcommittee then reviews the submission and makes a final recommendation to the federal health minister.

Problems with submissions were regarded as significant if both the evaluators and the technical subcommittee considered that the problem could have a serious bearing on the decisions made.


Most applications involved new drugs, or major changes to indication, conditions of use, or price. Of the 326 submissions, 279 (86%) were economic analyses based on randomised trials, with 238 (73%) containing direct comparisons of a new agent and a chosen comparator. Indirect comparisons with a common comparator were used in 41 (13%). Twenty-six (8%) were based on quasi-experimental designs and 21 (6%) were based on uncontrolled data (Table 1). Meta-analyses were used in 64 of the submissions.

Table 1: Types of studies used in economic evaluations

Design Number Percent
RCT direct comparison 238 73
RCT indirect comparison 41 13
Quasi-experimental 26 8
Uncontrolled 21 6
Total 326 100

Serious problems of interpretation were found in 218 (67%) submissions, and 31 had more than one problem, giving 249 serious problems in total. The main problems in these 218 submissions are shown in Table 2.

Table 2: Types of problem found in the 218 submissions that had problems

Problem Submissions Details Percent
Trial efficacy issues
Availability of trials 5
Poor quality trials 12
Interpretation of results 13
Use of surrogate outcomes 6
Determining therapeutic equivalence 26
Comparator issues
Uncertainty about choice of comparator or inappropriate comparator 6
Modelling issues
Technical aspects of the model 10
Unsubstantiated assumptions 6
Uncertainty about costs 13
Calculation errors
Errors introducing serious inaccuracies in estimation of cost-effectiveness ratios 4
Problems found with 218 submissions

Examples of problems encountered included:


None of these problems should be any surprise to regular readers of Bandolier, because they feature regularly in these pages as problems in assessing evidence. That pharmaceutical companies in Australia failed to recognise these issues in 1994-1997 should concern us for two reasons. The first is that most companies are now multinational and the second is that there was no improvement observed over the period of the study.

The authors are really quite gentle with the sponsors of the submissions, rightly identifying that pharmacoeconomic analysis is often a difficult and complex process. They do not believe the problems arise from any deliberate intent to deceive, but rather arose from a failure to take on board the requirements of quality evidence, process, and transparency.

Are things better now? It is difficult to know, without another analysis like this from Australia or elsewhere. Bandolier suspects that there remain huge lacunae of ignorance in the pharmaceutical world. A test is to use key words or phrases, like Cochrane, systematic review, NNT, or even Bandolier.

Will things change? You bet they will, because “fourth hurdle” issues alone will demand that change. Industry will learn that evidence-based medicine sells drugs, and they will find that it makes for good pharmacoeconomic arguments, too.


  1. SR Hill et al. Problems with the interpretation of pharmacoeconomic analyses. A review of submissions to the Australian Pharmaceutical Benefits Scheme. JAMA 2000 283: 2116-2121.

previous or next story