Signature March/April 2015 - 29

Training. While specific training needs
vary across regions, subject, and experience levels, there is evidence from this
study and past research showing that
better training is needed, both to help
bring more reviewers into the pool and
to make sure that editors trust and are
confident in using those reviewers.
While publishers want to develop
author services globally and build key
partnerships, there are a growing number
of publicly funded organizations in
emerging markets with an agenda to increase internationalization and visibility
of the research they fund. Author, reviewer, and editor training (especially
writing well in English) is critical to these
institutions' goals around visibility and
also serves to strengthen the standing of
publicly funded organizations as pivotal
research bodies within their regions.
This demand for training both at the
individual and institutional level creates
opportunities for publishers to provide
these services and become valuable partners for research institutions in emerging
and high-growth markets. In addition,
editing services like Editage or Edanz,
and/or third-party reviewer services, such
as Rubriq and Peerage of Science, could
have a role to play in helping to train
reviewers. This survey suggested that
few respondents currently have experience with third-party reviewer services,
with only 10 percent of respondents indicating they had provided reviews for
Peerage of Science, Academic Karma,
Rubriq, or Axios Review. The findings of
this survey suggest that training support
for reviewers is needed throughout the
researcher career arc, not just for those
new to reviewing.
Recognition. In many ways, training
and recognition/reward issues are different sides of the same coin. The drivers
for more effective reward and recognition
initiatives are a combination of the need
to adequately compensate reviewers for
the effort and time they take, a desire to
keep reviewers motivated, and a need to
reward the best reviewer attributes and
behaviors to maintain ongoing quality of
peer review standards.

Feedback from journal
editors is a vital form
of recognition for
reviewers.
However, the question remains: What
are reviewers being rewarded for -
simply completing a review, or completing a review well? Most journals monitor
quantitative aspects of reviewer performance: punctuality in delivering a report,
review invitation acceptance rate, number
of review reports delivered, and so on.
This is relatively easy to do as the numbers
are readily available and are indisputable.
However, measuring quality or usefulness
of a review is a different matter. A considerable number of journals have developed
frameworks for qualitative assessment of
reviewer performance, but reliably scaling
up this approach to the program level is
a challenge, relying on consistent execution of assessment standards from multiple
editors across multiple journals.
This survey suggested that the most
valued recognition initiatives are as much
about improving editorial workflows -
for example by telling reviewers how
useful their review was and sharing decision outcomes - as they are about receiving more formal compensation. In this
sense, feedback is itself a powerful form
of reward. Many journals share decision
outcomes and other reviews with their
reviewers, but how much more powerful
would it be if journal publishers could
harness this form of evaluation and apply
the same consistency in feedback as reviewers are asked to supply in the quality
of their comments?

CORE COMPETENCIES -
A POSSIBLE SOLUTION
Publishers need a consistent answer to the
question: What makes a good reviewer?
Studies have tried to quantify the charac-

teristics of a good reviewer (experience,
proven review frequency, etc.), but these
studies are primarily based on quantitative
factors. Alternatives, such as looking at
attributes based on the profile of current
reviewers, could just serve to reinforce the
existing distribution of reviewing effort.
One possible solution is the concept of
core competencies. If association publishers
want to train reviewers effectively and also
measure their performance, perhaps there
is a need to establish an industry-wide set
of minimum core reviewer competencies.
A set of universally agreed reviewer competencies, with some variation at subject level,
could provide the basis for both a training
framework and ongoing measurement and
evaluation of reviewer quality.
If there is a lack of trust in the reviewing ability of emerging and high-growth
market researchers, a training and recognition mechanism based on core competencies could help alleviate this issue. There
is an opportunity for centrally funded reviewer training programs, delivered by
publishers who have the expertise and the
content, tailored according to regional
needs, and designed to deliver learning
outcomes based on these competencies.
In 2014 alone, over 30 editorials in Wiley
journals offered guidance on peer reviewing. These were some of the most highly
viewed articles in the year, indicating a
strong researcher interest in more information and guidance on reviewing.
Finally, this same competency framework could also provide meaningful feedback on review activity to more established reviewers. The recently launched
"Think. Check. Submit" campaign
(thinkchecksubmit.org) is a good example
of what can be achieved by a coalition
across the field of scholarly communications. Further collaboration could greatly
facilitate progress on this issue also. n
Verity Warne (verity.
warne@wiley.com) is
associate marketing director
of peer reviewer engagement,
author marketing, for Wiley.

MARCH/APRIL 2016

signature

http://www.thinkchecksubmit.org

Table of Contents for the Digital Edition of Signature March/April 2015