“Because it is the Right Thing to Do”: Taking Stock of the Peer Reviewers’ Openness Initiative

The Peer Reviewers’ Openness Initiative (PRO) promotes the sharing of data and code. PRO signatories pledge to provide a full review only for manuscripts that publicly share data and code, or include a justification why sharing is not possible. Since the punitive element of this approach attracted criticism, we conducted a survey to assess signatories’ experiences with PRO. Contrary to the criticism, the reported experiences were predominantly positive, and 92% (117/127) of the signatories indicated that they would sign the initiative again today. Only 19 out of 127 respondents (15%) experienced negative reactions. Almost 50 respondents suggested ways in which PRO could be improved. We conclude that, from the signatories’ perspective, the benefits of the PRO initiative outweigh its drawbacks.

In recent years, several large-scale replication efforts suggested that the psychological literature is less reliable than many had hoped (e.g., Camerer et al., 2018;Klein et al., 2014Klein et al., , 2018Nosek & Lakens, 2014;Open Science Collaboration, 2015). Specifically, replication studies often fail to produce significant findings and generally yield effect sizes that are substantially smaller than those reported in the original studies. These disappointing results have arguably caused a "crisis of confidence" (Baker, 2016;Pashler & Wagenmakers, 2012;Spellman, 2015) that motivated the field to adopt more transparent research practices. One of these practices is the public sharing of data, materials, and analysis scripts (e.g., Chambers, 2017;Houtkoop, Chambers, Macleod, Bishop, Nichols, & Wagenmakers, 2018;Kidwell et al., 2016).
Although the practice of publicly sharing data and code appears relatively straightforward, it is still not the norm (e.g., Wicherts, 2013). In order to change the status quo and accelerate the adoption of data sharing practices, Morey and colleagues (2016) introduced the Peer Reviewers' Openness Initiative (PRO); researchers who sign PRO agree to provide a full review only for manuscripts that publicly share data and code, or else provide a clear reason why sharing is not possible.
PRO has enjoyed a mixed reception-the initiative has attracted criticism for being too strict, for being opposed to American Psychological Association (APA) guidelines, and for frustrating scientific progress (Bishop, 2016;Naik, 2017). Almost three years after PRO was launched, it remains unclear how the signatories themselves evaluate the initiative, whether they were able to live up to their own standards, and which challenges they have faced since signing PRO. Therefore, we sought to take stock of the initiative by surveying signatories on their subjective experiences and opinions. Specifically, we queried signatories' attitudes about the effectiveness of PRO, we asked about their positive and negative experiences since signing PRO, and we invited signatories to submit ideas for improvements. By giving signatories the opportunity to present their views and insights, this survey aimed to provide a more complete picture of the initiative, to possibly refute some of the early criticism, but also to spark a discussion on how to drive the initiative forward. This was an exploratory study.

D Disclosures
Data, Materials, and Online Ressources Readers can access the anonymized raw and processed data, the Qualtrics survey, and the R code needed to preprocess the raw data, and to generate the descriptive plots in our OSF folder at: https://osf.io/hxmqw/.

Reporting
We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

Method
At the time of data collection, 449 researchers had signed the PRO initiative. We successfully retrieved the email addresses of 340 signatories and these were sent the link to the Qualtrics survey. Compared to surveys on similar topics, the response rate of 37.65% (128/340) was relatively high (e.g., 4.99% for Houtkoop et al., 2018;4.32% for Schmidt, Geheimholzer, & Treloar, 2016). We excluded one respondent from the analysis because he or she only partially responded to the survey questions; therefore, the final sample size was N = 127.
The survey included six questions. 1 The first two questions were multiple-choice and were concerned with negative and positive experiences since signing the initiative (i.e., Have you had any of the following negative/positive experiences or reactions since signing the PRO Initiative? "Negative" answer choices: Being 'blacklisted' by a journal, Criticism from colleagues, Criticism from journals; "Positive" choices: Data was made publicly available, Praise from colleagues, I, as a reviewer, could provide improvements for the manuscript, e.g., accuracy of statistics). Multiple responses were possible. The remaining questions were partly open and inquired whether or not the signatories had shared their own data since signing the initiative (i.e., Since signing the PRO Initiative, have you shared data and code in your own manuscripts? If no, why?), whether they would sign the initiative again given their experiences so far (i.e., Given your experiences, would you still sign the PRO initiative today? If yes, why? If no, why not?), and whether they had any suggestions for improvements for the PRO initiative, or further comments. The authors Dahrendorf and Mittenbühler evaluated the open answers in an iterative process. First, each author grouped similar answers together to derive response categories. Then, the authors discussed which response categories would be included in their coding scheme and subsequently used this scheme to classify the signatories' responses. Any disagreement was resolved by discussion.
We polled the opinions of signatories without providing any reimbursement. Participation in the survey was entirely voluntary, and we indicated that the data would be analyzed anonymously. When the consent of the respondent was obtained, we included their answers to the open questions in our published data set. Respondents who participated in the survey agreed to make their anonymous data (including Dahrendorf et al.: Taking Stock of PRO their answers to the open questions) publicly available. After participating in the survey, however, we contacted three signatories again since we were concerned that their answers might disclose their identity. Therefore, we gave these signatories the possibility to reword their responses. The performed modifications exclusively concerned the grammatical structure of the sentences. In accordance with Dutch ethics-review procedures, we did not seek approval from an institutional review board for this nonmedical study (Central Committee on Research Involving Human Subjects, 2002). This study was conducted in accordance with the Declaration of Helsinki.

R Results
Regarding positive experiences with PRO, about 40% of the respondents indicated that data had been made available, about 30% reported to have received praise from colleagues and about 25% indicated to have been able to provide a higher-quality review ( Figure 1). In contrast, only a small fraction of respondents reported negative experiences, that is, being blacklisted by a journal or being criticized by colleagues or journals (Figure 2).
Commensurate with these experiences, 117 respondents indicated that they would sign the initiative again, whereas only eight indicated that they would not (multiple answers were possible; Figure 3).
As can be seen from Figures 1-3, signatories' experiences with the PRO initiative have been predominantly positive. From the signatories' perspective, the PRO initiative seems to be effective in promoting data sharing as well as improving the review process.  In addition, signing PRO is met with more positive than negative feedback from colleagues. For instance, one signatory stated: "Although I have been kicked off of reviews a few times, I have generally had positive responses from editors and have been asked to help write future journal policies". Yet, a few responses demonstrate that the PRO approach can have negative consequences. One of the concerns that has been repeatedly expressed in the literature is that the initiative might put too much pressure on authors and might be too coercive (e.g., Bishop, 2016;Naik, 2017). This coercive aspect could cause PRO to unwittingly exert an adverse effect on open science. Indeed, some signatories mentioned that the approach might be too aggressive and that the way of requesting authors to share their data might need to be changed (n = 11). One signatory stated, for instance: "I think there needs to be greater framing of the incentives/advantages of data sharing and open science practices, at the moment people could feel coerced or pressured". Whether signatories reported positive or negative experiences with PRO could partly depend on their career stage. As mentioned by four respondents, it may be relatively difficult for an early-career researcher to adhere to the PRO principles because the potential disadvantages (e.g., being unpopular with editors) are more severe.
In total, 49 signatories proposed improvements for the PRO initiative (the complete list of improvements mentioned in this survey can be found in the published data set accessible via https://osf.io/j3t6x/). The most common suggestion (n = 20) was to expand the reach of the initiative by various means (e.g., encouraging other open science initiatives and journal editors to sign PRO, more activity on Twitter, improving website/presentation). This might also address the issue reported by other signatories, namely that they simply forget to enforce PRO (n = 8). The majority of respondents, however, expressed their support for PRO: the most commonly stated reasons to still sign today was the belief in the initiative's contribution to science and its values and ideals (n = 59). Five respondents used the exact phrase: "the right thing to do". Another participant stated: "Because I totally agree with the values promoted by the PRO Initiative, whatsoever colleagues', editors', or journals' reaction." Finally, Figure 4 shows that 80% of respondents reported sharing (or at least sometimes sharing) their own data since signing the PRO initiative. 2 This demonstrates that respondents stand by the values they try to promote, as the percentage of researchers sharing data is noticeably higher than in the broader research community (Houtkoop et al., 2018). In some cases, participation in PRO might even motivate signatories themselves to share their data more readily, as explicitly mentioned by one respondent: "[The PRO initiative] forces me to apply the same standards to my research activities." Twenty-one researchers stated reasons for not 2 Note that due to an error in the encoding of this question by the authors, responses of four signatories could not be allocated to one sharing their data. Among the most common reasons, respondents mentioned restrictions due to sensitivity of data from the government, the institution, or the organization involved (n = 7), not having had opportunities to share data (n = 6) as well as not agreeing on data sharing policies with coauthors (n = 4). These reasons have also been found in the broader research community, but less commonly (compared to, for example, data sharing not being a common practice in the field, or fear of losing control over the data; Houtkoop et al., 2018).

C Conclusion
Signatories of the PRO Initiative pledge to provide a full review only for manuscripts that share their data or provide reasons why such sharing is impossible. In order to gauge the impact, public reception, and future prospects of the PRO initiative we conducted a survey among the PRO signatories. The survey revealed that experiences from PRO signatories were predominantly positive. Signatories reported that their adherence to PRO was generally met with approval, caused authors to make their data publicly available, and that it facilitated the review process. Furthermore, over 90% of the respondents indicated that they would sign the PRO initiative again today. Almost 50 respondents suggested ways in which PRO could be improved. A common suggestion was to promote the PRO initiative more energetically; one respondent suggested that members from of the answer options and thus were classified as ' 'No answer". other open science initiatives could be encouraged to sign PRO. We believe that this constitutes a promising strategy; for instance, PRO could be promoted through the Open Science Grassroots Community Network which currently encompasses over 200 initiatives from various disciplines (see https://tinyurl.com/y2l9gpxc for a crowdsourced list).
Only 15 % of respondents experienced negative reactions from journals or were criticized by colleagues. Some respondents expressed the concern that the initiative is too coercive-a criticism that has been voiced previously in the literature. With respect to data sharing, Houtkoop and colleagues (2018) reported that the most prevalent fears were that their data might get misinterpreted, they might get scooped, or that they might lose control over intellectual property. As a result, researchers might experience data sharing as a threat rather than an integral component of the scientific process. Thus, in order to increase the willingness to share data, the PRO initiative might need to address these concerns in greater detail. For instance, authors may be reminded that the presence of open data accelerates scientific progress and that data sharing can increase the impact of their work.
With these results in mind, it is important to note that this survey only concerns the experiences of a relatively small and highly selective sample. Therefore, one cannot draw general conclusions about the effectiveness and reception of the PRO initiative. Such conclusions necessitate the involvement of researchers on the receiving end of PRO, namely editors and authors. Regarding the signatories' positive and negative experiences, it should be noted that we kept the survey purposefully simple. In future years, should PRO find broader adoption, it may be possible to study more complex questions such as the impact of lab environment and scientific field.
Overall, the signatories' feedback shows that PRO has potential pitfalls, can meet resistance, and shows room for improvement. But at the same time, in its almost three-year existence the signatories are predominantly positive about PRO and its ability to contribute to a more open and transparent science. At least from the signatories' perspective, the PRO initiative has demonstrated that it is a reasonable approach to improving science.
A Author Contributions MD, TH, MM, and SW jointly generated the idea for the study, collected the data, and wrote the first draft of the manuscript. TH wrote the analysis code and analyzed the data, and AS verified the accuracy of those analyses. AS, DM, and EJW revised the manuscript. All authors approved the final submitted version of the manuscript.