Baylor University Medical Center Proceedings April 2017 - 164


studies (11). Two screeners independently reviewed the titles
and abstracts of each citation and made a decision regarding
its suitability for inclusion based on the deﬁnitions previously
described. To standardize the coding process, an abstraction
manual was developed and pilot tested. After completing this
process, a training session was conducted to familiarize coders
with abstracting the data elements. A subset of studies was jointly coded. After the training exercise, each coder was provided
with 3 new articles to code independently. Inter-rater agreement
of these data was calculated using Cohen's kappa. Since interrater agreement was high (k = 0.86; agreement = 91%), each
coder was assigned an equal subset of articles for data abstraction. We coded the following elements: a) statistical test used
to evaluate heterogeneity; b) a priori threshold for statistical
signiﬁcance; c) type of model (random, ﬁxed, mixed, or both);
d) whether authors selected a random-eﬀects model based on
signiﬁcance of the heterogeneity test; e) whether authors used
a random-eﬀects model without explanation; f ) what type of
plot was used to evaluate heterogeneity, if any; g) whether the
plot was published as a ﬁgure in the manuscript; h) whether a
follow-up analysis was conducted and, if so, the type of analysis (subgroup, meta-regression, and/or sensitivity analysis); i)
whether heterogeneity was mentioned in writing only; and j)
whether authors concluded there was too much heterogeneity to
perform a meta-analysis. After the initial coding process, validation checks were conducted such that each coded element was
veriﬁed by the other coder. Next, the screeners held a meeting to
discuss the diﬀerences in decisions for inclusion/exclusion and
reconcile any discrepancies by reaching consensus. Following
the screening process, full-text versions of included articles were
obtained via EndNote. Analysis of the ﬁnal data was conducted
using STATA 13.1. Data from this study are publicly available
on Figshare (http://dx.doi.org/10.6084/m9.ﬁgshare.1496574).
RESULTS
The PubMed search resulted in 337 articles from 6 journals.
After screening titles and abstracts via Covidence, 79 articles
were excluded that did not meet the deﬁnition of a systematic
review and/or meta-analysis. Full-text article screening for exclusion criteria resulted in the removal of an additional 74 articles.
Additionally, 2 studies could not be retrieved. Two of the 6
journals were not heavily represented in the original sample of
337 articles, and with the exclusion of 155 articles 2 journals
were excluded from the ﬁnal sample. In total, 182 manuscripts
representing 4 journals were analyzed for heterogeneity (Figure).
Half (91/182) of all meta-analyses used at least 1 heterogeneity test. The most widely reported statistic was I2 (41.2%;
75/182) followed by X2 (24.2%; 44/182). In combination,
X2 and I2 (13.2%; 24/182) were reported with greatest use
followed by Q and I2 (12.6%; 23/182). Other combinations
were utilized by the manuscripts but were not used to a great
extent (I² and Tau² [0.55%; 1/182]; Q and X² [0.55%; 1/182];
Q, X², I² [3.3%; 6/182]; and X², I², Tau² [0.55%; 1/182]). As
shown in the Table, authors selected a random-eﬀects model
most frequently (25%) followed by both ﬁxed- and randomeﬀects models (21%). Fixed-eﬀects models were reported in 4%
164

Figure. Preferred Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) flow diagram of study selection.

of studies, and a mixed-eﬀects model was used in only 1 study.
The remaining 48% did not report the type of model used for
analysis. Twenty-four percent (43/182) used the random-eﬀects
model without considering the results of a heterogeneity test
to conﬁrm the need for such an analysis, and 15% (27/182)
changed from the ﬁxed- to random-eﬀects model based on the
results of a heterogeneity test.
The level of statistical signiﬁcance for heterogeneity tests
was reported in 45 systematic reviews. Among those reporting
predeﬁned thresholds for statistical signiﬁcance, the most frequently reported P value was < 0.05 (64.4%; 29/45) followed
by P < 0.10 (31.1%; 14/45). (P < 0.01 and P < 0.001 were both
reported in 1 study.) Forty-three percent (78/182) of systematic
reviews contained heterogeneity plots published as ﬁgures in the
article (Table). A forest plot was the most common heterogeneity
plot (42%). Only 2% used an L'Abbé to graphically represent
heterogeneity.
Of the 3 tests designed to investigate heterogeneity (subgroup, meta-regression, and sensitivity analyses), subgroup
analysis was used the most (21%), sensitivity analysis was second
(18%), and meta-regression was used the least (9%) (Table 1). It
was found that 20% (36/182) of the available manuscripts wrote
about heterogeneity, but never actually evaluated it. Fifty-eight
percent (105/182) of manuscripts did not ﬁnd signiﬁcant heterogeneity, 3% (5/182) found enough evidence of heterogeneity
to disregard "some" of the meta-analysis, 4% (8/182) found
signiﬁcant heterogeneity, and 35% (64/182) never attempted
to assess heterogeneity.

Baylor University Medical Center Proceedings

Volume 30, Number 2
http://dx.doi.org/10.6084/m9.figshare.1496574
Table of Contents for the Digital Edition of Baylor University Medical Center Proceedings April 2017
