The Benefits and Limitations of Preregistration and Registered Reports

We started the event with a quick poll. It was interesting to see that almost everyone has heard of Preregistration and Registered Reports already, but only two have done that. Perhaps more of them have already written a data management plan or asked for ethical approval. We will see later that the two are not that far away from Preregistration.

Figure 1: Answers to the question: Have you ever heard of Preregistration or Registered Reports?

Figure 2: Answers to the question: Have you ever preregistered a study or written a Registered Report?

What is actually the problem that makes something like Preregistration or Registered Reports necessary? Well, there are several so-called questionable research practices (QRP) (Bouter 2019), such as

• HARKing - Hypothesising After Results are Known
• p-hacking - tuning the data analysis to achieve significant p-values
• selective reporting - reporting only on positive results ("cherry-picking")

Are these practices intentional? Let's be optimistic and assume these practices happen unintentionally, e.g., due to wrong or missing education, a lack of standards in the research area, or pressure from supervisors, reviewers, and editors. There are also some further drawbacks related to the scientific system:

• Publication bias is the phenomenon in which studies with positive results are more likely to be published than studies with negative results
• Due to the late review process, researchers receive feedback too late

Some of the problems listed above can be solved through Preregistration. But what is Preregistration? Nosek et al. (2018) stated:

Preregistration includes the following elements (Wicherts et al. 2016): First, metadata and whether we want to generate or test hypotheses. In the latter case, we need to include a list of hypotheses or research questions. Next, we need to add details regarding the study design, e.g., sample size, measured variables, experimental conditions, and, finally, the analysis plan. Preregistration is usually done after designing the study and before data collection. It does not require special permission from or connection to a journal. All current authors involved in the project should approve the Preregistration before submitting it. Preregistration receives a timestamp (publication or submission date) and a DOI, making it citable. A Preregistration is not necessarily public but can be embargoed for up to a few years. By the way, qualitative research can be preregistered as well, and there is even a particular form for that (Haven and Grootel 2019). Two popular platforms provide forms to facilitate Preregistration: Open Science Framework and AsPredicted.

Coming back to our questionable research practices. Which of these can Preregistration solve? First, indicating whether hypotheses are generated or tested makes HARKing impossible. Second, providing details regarding the analysis plan prevents p-hacking. Finally, listing the hypotheses makes it hard to hide unfavourable or undesired findings. Awesome! But to which problems does Preregistration not contribute? Publication bias is still possible since Preregistration is independent of journals, and feedback from reviewers still comes too late.

So Preregistration apparently solves quite a few issues, and the forms provided by the platforms make it very easy. Does that mean that a preregistered study is an indicator of good quality? Unfortunately not. Sanjay Srivastava asked his students to compare preregistrations with what was reported in the final paper. The students found quite a few discrepancies between preregistrations and the final paper. For example, the Preregistration only included a subset of the analysis reported in the final paper. In some cases, essential elements were missing. For example, the analysis plan or the reported design was too vague.

These observations show that a preregistered study is not necessarily well done. One reason is that Preregistrations are usually not peer-reviewed, and this is where registered reports can help.

Before we dive into Registered Reports, let's first check the research process. In a nutshell (Figure 3), researchers usually start developing the idea, generating hypotheses, and formulating research questions. To address these, researchers design a study and start collecting data which is then analysed. The findings are reported in a paper and submitted for peer review. If it gets accepted, it is published and read by others and used to develop new ideas. However, in my personal experience, papers usually get rejected. At best, because some paragraphs are unclear. At worst, because the results are not significant, more data is needed, the study is not appropriate, or the research's novelty is unclear, requiring to start from scratch.

Figure 3: Research process

Similar to Preregistration, Registered Reports are also written after designing the study and before data collection. However, Registered reports are almost scientific articles as they contain a full introduction, literature review, methods, and an analysis plan. The report is then submitted to a journal and peer-reviewed. This, of course, changes the research process dramatically. A research process based on Registered Reports is composed of a two-stage review process (Figure 4) (Chambers 2019). The first peer review is happening on the Registered Report (Stage 1).

In contrast to Preregistration, a Registered Report is consequently not independent of a journal. If it gets rejected, the researcher either needs to re-formulate the hypothesis or change the study design. If it gets accepted, data collection can start. Hence, a successfully registered report is equivalent to a guaranteed publication after acceptance independent of positive or negative results (In-principle acceptance). In Stage 2, the entire paper will be reviewed. The reviewers check whether the researcher did what was promised. If this is the case, the paper will be published. It is a common misunderstanding that it is not allowed to deviate from the study design described in the Preregistration or Registered Report. It is possible, but researchers need to argue about why they changed the procedure.

Figure 4: Two-stage review process based on Registered Reports.

The benefits of registered reports are summarised in the following two quotes.

Coming back again to the problems we listed at the beginning. In addition to the benefits from Preregistration, Registered Reports also help avoid publication bias since the acceptance of a manuscript does not depend on positive or negative results. And researchers receive very early feedback from reviewers. However, there are also several limitations. A Registered Report requires careful time management and planning due to the review time needed at stage 1.

Furthermore, Registered Reports are not recommended if data collection has to start soon or if grant deadlines are close. It might also be challenging to combine Registered Reports with student projects which usually run for a few months. A significant issue is that neither Preregistration nor Registered Reports are commonly provided by journals or conferences. Finally, researchers should not underestimate the effort needed to create a well-thought-out analysis plan. However, in the end, it is more of a shifting of tasks you would do anyway more to the beginning of the research process (Figure 5).

Figure 5: By Center For Open Science: https://www.cos.io/initiatives/registered-reports

To sum up, Preregistration and Registered Reports can help tackle questionable research practices and even some of the scientific system's drawbacks. Still, both concepts are relatively new and not well-established across all scientific disciplines. Table 1 summarises the characteristics of Preregistrations and Registered Reports.

We concluded the event with a little roundtable discussion. Peter de Vries from the Faculty of Behavioural, Management, and Social Sciences (BMS) started by mentioning one of the key benefits:

However, on the downside,

Steven Watson (BMS) replied that this might be a problem for Preregistration but not so much for Registered Reports which take a lot of effort to prepare them. He also considered that

Raúl Zurita-Milla shared some insights from ITC where students need to write and defend a proposal for their thesis, which could be used easily to extract Preregistration.

Another issue was raised by Stéphanie van den Berg. In her opinion, there is too much hypothesis testing in areas where it is not useful. She stated that this is also one of the main reasons for issues like p-hacking. Steven agreed and added that Preregistration does not tell much about the research quality, e.g., if the selected model is appropriate.

Two persons from the poll indicated that they have experience with Preregistration. Fortunately, they were willing to share their experiences. Peter de Vries preregistered a replication project. It took him just one hour and was even reviewed. He would do it again since it is a valuable addition, so he also encourages his students to do so. Steven added that Preregistration “is probably the easiest way to start” with Open Science. It is also easily possible to change the analysis plan. All you need to do is justifying the changes, for example, by pointing to reviewers’ suggestions. This need for justification can also make flawed reasoning more visible.

Peter Noort (LISA) preregistered a qualitative literature review and experienced the following:

Thanks to all for attending the event and for the great contributions. We are already looking forward to the next Open Science Kitchen event, which will take place on the 25th of March. Stay tuned, spread the word!

By Markus Konkol

Slides

This video player uses cookies

The YouTube player uses marketing cookies. Accept third-party cookies to load the player. Watch without cookies on the YouTube website.

References

Bouter, L. (2020). What Research Institutions Can Do to Foster Research Integrity. Science and Engineering Ethics, 26(4), 2363–2369. doi:10.1007/s11948-020-00178-5

Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, David T. Mellor (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115 (11) 2600-2606; https://doi.org/10.1073/pnas.1708274114

Chambers, C. (2019). What’s next for Registered Reports? Nature, 573(7773), 187–189. doi:10.1038/d41586-019-02674-6

L. Haven, T., & Van Grootel, D. L. (2019). Preregistering qualitative research. Accountability in Research, 26(3), 229–244. doi:10.1080/08989621.2019.1580147

Quintana, D. S. (2020). Five things about open and reproducible science that every early career researcher should know. https://doi.org/10.17605/OSF.IO/DZTVQ

Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.01832