Critical appraisal
The process of assessing and interpreting evidence by systematically considering its validity, results and relevance.
Bandolier has available a longer essay on critical appraisal - 
What is critical appraisal?
This sounds great, and often is. Of course, one needs to look at different forms of evidence differently - controlled trials, versus observational studies, for instance. Or diagnostic studies, or health economic studies. A number of schema have been developed for doing this, and are useful. The trouble is that any one of them can be wrong for a given paper, because some things are hard to define, like what constitutes a valid or relevant study.
The three things to bear in mind are quality, validity, and size:
- Quality: trials that are randomised and double blind, to avoid selection and observer bias, and where we know what happened to most of the subjects in the trial.
- Validity: trials that mimic clinical practice, or could be used in clinical practice, and with outcomes that make sense. For instance, in chronic disorders we want long-term, not short-term trials. We are not interested in small but marginally statistically significant (p < 0.05, say, or a 1 in 20 chance of being wrong), but outcomes that are large, useful, and statistically very significant (p < 0.01, a 1 in 100 chance of being wrong).
- Size: trials (or collections of trials) that have large numbers of patients, to avoid being wrong because of the random play of chance. For instance, to be sure that a number needed to treat (NNT) of 2.5 is really between 2 and 3, we need results from about 500 patients. If that NNT is above 5, we need data from thousands of patients.
These are the criteria on which we should judge evidence. For it to be strong evidence, it has to fulfil the requirements of all three criteria.