Identity, Universalism and Peer Review (Peer Review Week 2021)

Peer review is the foundation of the scholarly publication system, at the heart of the system of science. It’s an essential cog in the scientific machine. The process involves the subjecting of scientific papers, which detail the methods, results and claims of new research, to expert scrutiny. This critical dissection is carried-out by scientists with similar competencies and expertise to that of the authors. It’s a time-consuming, challenging (for both author and reviewer) and, as most scientists will attest, often-stressful hurdle on the journey from lab to publication. It affords us - scientist and/or citizen - some confidence that the findings and claims made in scientific publications are credible, valid, trustworthy, and based on sound methodology. It ensures that the evidence that is used to inform policy decisions is veracious and reliable. But it is necessary to ask whether this much-celebrated backbone of science is operating at its best. Considering this year’s Peer Review Week theme, identity, I’m asking how the identity of authors and reviewers impacts the peer review process, and the trustworthiness and quality of scientific publications.  

Dr Robert Merton stated that scientific claims should be subjected to ‘pre-established impersonal criteria’, regardless of the ‘personal or social attributes of their protagonist’ [1]. He referred to this as ‘universalism’. This value is widely accepted as a core scientific imperative, at least in principle, if not in practice. Within the context here, it means that the demographic characteristics of the authors (e.g., nationality, ethnicity, gender, or institutional affiliation) should not influence the reviewer’s objective assessment of the truth-claims contained within a scientific paper. Instead, these claims should be judged on the merits of the scholarship, divorced from considerations of who carried-out the research. However, humans are biased. We make judgements based on our pre-conceived ideas, cognitive short-cuts, and our implicit categorisations of others. And contrary to scientists’ commitments to impartiality and efforts to remain objective, such biases influence various components of the scientific system. For example, a 2012 study found that male applicants for a laboratory manager position were rated as significantly more competent than female applicants, with an otherwise identical application [2]. Moreover, a JAMA study found that a sample of Clinical Orthopaedics and Related Research journal peer reviewers recommended accepting articles containing the name of a prominent researcher from a prestigious institution at a rate of 87 percent, compared to 68 percent for otherwise identical articles but with the authors’ names redacted [3].

But, a simple remedy exists. Academic journals can mask the identities of authors from reviewers by removing their names and institutional affiliations. A variant of this practice is already commonplace within the scholarly publication system. However, typically it only involves the masking of the reviewers’ names from the authors and not the other way around. (Note, this practice, known as ‘single anonymous peer review’ (or ‘single blinding’), has advantages too. For example, it affords the reviewer the ability to make critical comments without discomfort or fear of reprisal – a practice especially beneficial to early-career researchers (ECRs) reviewing the work of esteemed colleagues.) Yet, single anonymous peer review is only a short step away from full double anonymous peer review (or, ‘double blinding’). That is, the practice of additionally masking the authors’ identities from the reviewers. Implementation of this practice by academic journals will reduce the influence of reviewer’s biases when judging truth-claims made in scientific papers. While scientists should be trusted and encouraged to address the biases that we are all susceptible to, such a simple move could potentially reduce the likelihood that truth-claims in scientific papers are judged on irrelevant criteria that have nothing to do with the quality or merit of the scholarship. Of course, it’s not a complete fix – for example, it won’t stop reviewers dismissing and sabotaging claims that run counter to their own paradigmatic or theoretical standpoints, or socio-political viewpoints. However, double anonymous peer review will go some way in realigning scientific practice with the values of impartiality and objectivity to which scientists pay lip service.

We additionally need to diversify and grow the pool of peer reviewers, especially by including those from underrepresented groups, like women, ECRs and ethnic minorities. If reviewers are predominantly white, male, and late career, the insights, perspectives, and experiences represented will be reflective of these narrow demographics. This applies to viewpoint diversity too. Additionally, a small and non-diverse reviewer pool limits the availability and scope of expertise that editors can call-upon, meaning the most relevant and appropriate expertise that exists out there will fail to be utilised. In considering the goal of science more broadly, the exclusion of these individuals from the scientific process means that potentially worthwhile contributions to the field, some potentially world-changing, may never become known. However, we should be cautious of mandates and quotas that require equal representation of reviewers from underrepresented groups, as this may disproportionately burden those that are underrepresented, given that they potentially represent a minority in the system more generally. Demographic and viewpoint diversity are essential in peer review, as they are with the system of science more generally. As the number of submissions to academic journals increase, these issues are likely to become more pronounced, and efforts to grow and diversify the pool of peer reviewers becomes ever more important.

Initiatives and approaches aimed at diversifying the scientific enterprise, particularly the scholarly review process, should be multifaceted and include a multitude of stakeholders. Taking ECRs as a key example (a group that constitute a large fraction of the scientific community), ensuring that they recognise peer review as a significant scholarly contribution, while providing them with the training to effectively review scientific papers is a key first step towards the goal of maximising engagement. Research training institutions have a role to play here, through integrating training into PhD and postdoctoral programmes. Efforts from charities such as Sense about Science, with their peer review training workshops and online resources for ECRs, are hugely beneficial. Credit should be afforded to ECRs undertaking peer review as a signal to journal editors that ECRs are skilled, and capable of effectively reviewing scientific manuscripts to a high standard. Reviewer mentoring schemes, such as the PRE-review Open Reviewers programme piloted in 2020, that involve the co-reviewing of a paper, between an ECR and a more-experienced colleague, will no doubt be part of this solution. And of course, journals and their editors should simply be inviting more ECRs to engage in the process.

These are not the only issues that pervade the system of science, nor the scholarly publication system more specifically. I additionally do not seek to make any judgements as to the relative magnitude of the issues aforementioned or those omitted. Certainly, there are broader debates to be had on the utility of peer review and its effectiveness in picking up errors in scientific research, but these are for another time. As a final caveat, and to avoid readers from concluding that science is not up to the job, the system of science is marvellous, brilliant and hugely effective. It’s undoubtedly humanity’s greatest achievement, and while it’s necessary to tease out the kinks, it’s the best thing we have to identifying truth, accruing knowledge and saving lives.

[1] Merton R., & Storer, N. (1973). The Sociology of Science: Theoretical and Empirical Investigations, University of Chicago Press.

[2] Moss-Racusin, C. A., Dovidio J. F., Brescoll V. L., Graham M. J., & Handelsman J. (2012). Science faculty’s subtle gender biases favor male students. PNAS, 109(41), 16474–9. https://doi.org/10.1073/pnas.1211286109 

[3] Okike, K., Hug, K. T., Kocher, M. S., & Leopold, S. S. (2016). Single-blind vs double-blind peer review in the setting of author prestige. The Journal of the American Medical Association, 316(12), 1315-1316. doi:10.1001/jama.2016.11014