Baylor University Medical Center Proceedings April 2017 - 165

Table. Descriptive heterogeneity practices
Cancer
Research
(n = 1)

Clinical Cancer
Research
(n = 17)

Journal of
Clinical Oncology
(n = 106)

The Lancet
Oncology
(n = 58)

Total
(n = 182)

76 (42%)

Forest plot, L'Abbé plot

3 (2%)

Forest, L'Abbé, Galbraith plot

0 (0%)

None

103 (57%)

Subgroup analysis

39 (21%)

Meta-regression

17 (9%)

Sensitivity analysis

33 (18%)

Both

39 (21%)

Fixed

8 (4%)

Random

46 (25%)

Mixed

1 (<1%)

None

88 (48%)

Category

Variable

Types of heterogeneity plots reported

Forest plot

Types of analyses reported

Choice of model reported

DISCUSSION
Systematic reviews operate based on methods designed
to assist researchers in minimizing bias, performing literature
searches, and evaluating data in a manner that hopefully limits
biased results. One important aspect of this systematic process
is the analysis of heterogeneity at its origin and the subsequent
eﬀect on meta-analysis. Although some studies assessed heterogeneity, it was not a common practice in the studies included in
this systematic review. Only half of the available studies applied
one or more of the common heterogeneity tests evaluated in this
study, with 20% of available studies mentioning heterogeneity
without further assessment. With interstudy variance always
present on some level, heterogeneity evaluation in systematic
reviews becomes necessary in many cases. The paltry use of
meta-regression and/or subgroup analysis (9% and 21%, respectively) limits studies assessing the eﬀect of heterogeneity
on meta-analysis. The random-eﬀects model was underused,
with 25% using this model and 21% using both random- and
ﬁxed-eﬀects models. The random-eﬀects model assumes that
parameters underlying studies follow a distribution, while ﬁxedeﬀects models assume a single parameter value common to all
studies (12). The random-eﬀects model is a more likely event
and may be used in cases of heterogeneous study outcomes. It
is recognized that all systematic reviews will have some level of
heterogeneity, and this study recommends the use of randomeﬀects modeling for meta-analysis of heterogeneous intervention
eﬀects. The Institute of Medicine's Standards for Systematic
Reviews state that "although the committee does not believe
that any single statistical technique should be a methodological standard, it is essential that the SR [systematic review] team
clearly explain and justify the reasons why it chose the technique
actually used" (13). From this review, only 15% of available
studies used the random-eﬀects model with justiﬁcation. This
ﬁnding highlights the need for greater explanation during the
decision-making process.
April 2017

There have been recent developments for exploring and interpreting heterogeneity both before and after the review process. Evidence mapping is a process developed for systematic
reviewers to explore sources of heterogeneity among primary
studies prior to pooling (14). This qualitative approach may be
a useful mechanism to identify sources of heterogeneity and
could be a mechanism to inform subgroup analyses. A second
means for interpreting heterogeneity is to calculate and report
prediction intervals. Prediction intervals may be used to present
an expected range of true eﬀects and can assist in the clinical
interpretation of heterogeneity by estimating an expected true
treatment eﬀect in future settings (15). We recommend that
these areas be explored in future research.
This study has many positive attributes, including its adequate sample size and the careful application of coding procedures. Additionally, our ﬁndings shed light on current practices
of heterogeneity assessment, which is greatly lacking in oncology
reviews. Given that little research on heterogeneity practices has
been conducted to date, especially in clinical specialties such
as oncology, comparison of our results with other studies is
diﬃcult. To our knowledge, this is the ﬁrst study of its kind in
oncology. This study also has some limitations. We examined
systematic reviews published in high-impact-factor oncology
journals, and our results may not represent oncology systematic reviews as a whole. It is possible that higher-impact-factor
journals have more rigorous reporting and methodological standards, and systematic reviews published in these journals may
reﬂect these standards. Our search was also date limited, and
our results should not be generalized outside of our search dates.
1.
2.

Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of
Interventions. Chichester: Cochrane Collaboration, 2008.
Thorlund K, Imberger G, Johnston BC, Walsh M, Awad T, Thabane L,
Gluud C, Devereaux PJ, Wetterslev J. Evolution of heterogeneity (I²)
estimates and their 95% conﬁdence intervals in large meta-analyses. PLoS
One 2012;7(7):e39471.

Heterogeneity of systematic reviews in oncology

165

Table of Contents for the Digital Edition of Baylor University Medical Center Proceedings April 2017