Baylor University Medical Center Proceedings April 2017 - 163

Heterogeneity of systematic reviews in oncology
Jonathan Holmes, BS, David Herrmann, BS, Chelsea Koller, BS, Sarah Khan, BS, Blake Umberham, BS,
Jody A. Worley, PhD, and Matt Vassar, PhD

Systematic reviews synthesize data across multiple studies to answer a
research question, and an important component of the review process
is to evaluate the heterogeneity of primary studies considered for inclusion. Little is known, however, about the ways that systematic reviewers
evaluate heterogeneity, especially in clinical specialties like oncology. We
examined a sample of systematic reviews from this body of literature
to determine how meta-analysts assessed and reported heterogeneity. A PubMed search of 6 oncology journals was conducted to locate
systematic reviews and meta-analyses. Two coders then independently
evaluated the manuscripts for 10 different elements based on an abstraction manual. The initial PubMed search yielded 337 systematic
reviews from 6 journals. Screening for exclusion criteria (nonsystematic
reviews, genetic studies, individual patient data, etc.) found 155 articles
that did not meet the definition of a systematic review. This left a final
sample of 182 systematic reviews across 4 journals. Of these reviews,
50% (91/182) used varying combinations of heterogeneity tests, and
of those, 16% (15/91) of review authors noted excessive heterogeneity
and opted to not perform a meta-analysis. Of the studies that measured
heterogeneity, 51% (46/91) used a random-effects model, 7% (8/91)
used a fixed-effects model, and 43% (39/91) used both. We conclude
that use of quantitative and qualitative heterogeneity measurement tools
are underused in the 4 oncology journals evaluated. Such assessments
should be routinely applied in meta-analyses.

ystematic reviews bring together all related empirical
evidence based on predetermined eligibility criteria to
answer a research question (1). This methodology is designed to minimize bias using an explicit, reproducible
approach involving a systematic and comprehensive literature
search, an assessment of validity of primary studies, and a systematic presentation and synthesis of ﬁndings. Oftentimes, systematic reviews also contain one or more meta-analyses that make
use of statistical procedures to summarize the results of primary
studies. It is evident that when multiple studies are combined
for data synthesis, there will be diﬀerences, such as location of
testing, drug doses, dosing schedules, follow-up, or ethnicity of
participants, to name a few. If statistically signiﬁcant heterogeneity is present, then researchers must decide if the primary studies
are too diverse to synthesize or if follow-up analyses should be
used to explore the eﬀects of these diﬀerences on study outcomes.
Proc (Bayl Univ Med Cent) 2017;30(2):163-166

Exploring heterogeneity between primary studies in
systematic reviews can be done with multiple statistical tests
such as I², Cochran's Q (chi-squared), and Tau². All of these tests
have their own strengths and weaknesses, and so it is best to use
multiple tests to fully inform clinicians on the dependability of
a systematic reviews analysis (1-9). While much advice has been
oﬀered on evaluating heterogeneity, little is known about the
ways that systematic reviewers actually address heterogeneity.
Questions remain regarding the practices of systematic reviewers outside of Cochrane review groups, such as researchers in
clinical specialties like oncology.
METHODS
Using the h5-Index from Google Scholar Metrics, we selected
the 6 oncology journals with the highest index scores from the
oncology subcategory. We searched PubMed using the following
search string: (((((("Journal of clinical oncology: oﬃcial journal of the American Society of Clinical Oncology" [Journal]
OR "Nature reviews. Cancer" [Journal]) OR "Cancer research"
[Journal]) OR "The Lancet. Oncology" [Journal]) OR "Clinical
cancer research: an oﬃcial journal of the American Association
for Cancer Research" [Journal]) OR "Cancer cell" [Journal])
AND ("2007/01/01" [PDAT]: "2015/12/31" [PDAT]) AND
"humans" [MeSH Terms]) AND (meta-analysis [Title/ Abstract]
OR systematic review [Title/Abstract]). This search strategy was
adapted from a previously established method that is sensitive to
identifying systematic reviews and meta-analyses (10). Searches
were conducted on May 18 and May 26, 2015.
We used Covidence (covidence.org) to initially screen
articles based on title and abstract. To qualify as a systematic
review, studies had to summarize evidence across multiple studies and provide information on the search strategy, such as search
terms, databases, or inclusion/exclusion criteria. Meta-analyses
were classiﬁed as quantitative syntheses of results across multiple
From the Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma
(Holmes, Herrmann, Koller, Khan, Umberham, Vassar) and the University of
Oklahoma, Norman, Oklahoma (Worley).
Corresponding author: Matt Vassar, PhD, Department of Psychiatry and
Behavioral Sciences, Oklahoma State University Center for Health Sciences,
1111 West 17th Street, Tulsa, OK 74107 (e-mail: Matt.vassar@okstate.edu;
Twitter handle, @bmattvassar).
163

http://www.covidence.org

Table of Contents for the Digital Edition of Baylor University Medical Center Proceedings April 2017