Multiple systematic reviews on the same question: some considerations
Once a clinical question has been clarified, it is useful to search the literature to check if there are any existing systematic reviews that have already attempted to answer the same or a similar question. However, if multiple reviews are identified, which one should be used?
Are two reviews better than one?
When assessing the best available evidence it’s not unusual to find multiple systematic reviews examining the same or similar clinical questions. It is to be hoped that the reviews reach similar conclusions, as this can give us confidence in those conclusions. But we do need to be careful not to overestimate the strength of evidence.
Often, reviews may have retrieved and analysed all or most of the same RCTs, so conclusions may be based on the same primary data. Before drawing any conclusions, check the primary data the reviews analyse.
What if the reviews disagree?
A more difficult situation arises when the reviews reach different, or even discordant, conclusions for the same question.This can occur because of different search methods, inclusion criteria, or analysis methods.
Which review should you believe? Begin by asking the following questions:
- Are the reviews valid, or do they contain methodological flaws? You can use the checklist for appraising systematic reviews to help you decide this.
- Is the difference between the reviews clinically important — i.e., would they cause you to make a different clinical decision? (e.g., is the direction of the effect similar, even if the size of the effect differs?)
- Are the reviews asking the same question? They might actually be looking at different aspects of a similar question: for example, they may be looking at different subgroups within a population, different outcomes, or different related interventions (e.g., different drugs in a drug class, or different psychological treatment protocols). In this case you should look at which review is most relevant to your particular question.
- Do the reviews identify the same or different primary studies? This will depend on the reviews’ search methods, their inclusion criteria, and also when they carried out the search. If the reviews identified different RCTs, think about why this was. In general, if both reviews are valid, use appropriate selection criteria, and ask the same question: the review with the most comprehensive search and based on the most data may be least likely to be biased.
The general rules when choosing which review to use
- If two (or more) reviews retrieve the same RCTs (or one reports the same RCTs as all the others plus a few more), the same outcomes are reported in the same population, and the review reaches the same conclusions, you might choose to use only the most recent review (by search date), and supersede all older reviews.
And the exceptions:
- If there are very few data available to answer the clinical question, you might choose to consider all the systematic reviews found.
- If the reviews identify the same RCTs, but an older review has superior methods to a more recent review, (e.g., it performs an appropriate and clinically useful meta-analysis or provides additional data on the RCTs while the later review does not), it might be appropriate to consider all reviews, taking into account the higher quality methods of the older review.
- If the reviews come to different conclusions and the difference is not explained by the addition of material subsequent to the older review, or by superior methods in the later review, it might be helpful to consider both reviews and also attempt to think about possible explanations for the difference in conclusions.
- If the RCTs retrieved by the reviews are the same, but the reviews performed different analyses (of outcomes or populations), it might be useful to consider both reviews and the overlap between them.
- If (a common scenario) the reviews contain some but not all of the same RCTs, it might be worth considering several reviews. But an older review can be superseded if it contains RCTs excluded by a later review on appropriate methodological grounds. However, if there doesn’t seem to be a good reason for the inclusion of different RCTs in reviews on the same question, or if each review gives different information, it might be appropriate to use multiple reviews. Understanding the relationships between these reviews is helpful, to make clear that you are not double-counting RCTs (and thus overestimating the strength of the evidence).
In conclusion
Although ideally you would wish to find a single, recent, high-quality review to answer a clinical question, there are many cases where you will need to consider several reviews and think about their relative merits.
Related BMJ Blogs
How much health evidence is there in the world?
How systematic reviews can reduce waste in research