TReQ: 2 Pre-registration

[Introduction slide 1: title “TReQ: improving the transparency, reproducibility and quality of your research”]

[Introduction slide 2: title “Video Two: Pre-Registration of Studies”]

Gesche Huebner: In this video I will describe what pre-registration is. I will start off by explaining the details of pre-registration, describe the benefits it brings, discuss any concerns around its application, and then finish off with some practical tips, how to actually use it.

[Slide titled “What does pre-registration mean?”]

Gesche Huebner: So first of all, what does pre-registration mean?

[Definition slide with title “Pre-Registration: describing the details of your study before you actually conduct the study”]

Gesche Huebner: Pre-registration means to describe the details of your study before you actually conduct the study.

[Text slide with bullet points:

• What is the target population?
• What is your sample size?
• What statistical tools will you be using?
• What is your research question?
• What are your hypotheses where applicable?]

Gesche Huebner: So for example, what is the target population? What is your sample size? What statistical tools will you be using? What is your research question? And what are your hypotheses where applicable? You write all these details into a document and upload an online registry. You will then be given a timestamp and a registration number. You can make this pre-registration, also called a pre-analysis plan, immediately available to others to find, or you can keep it private until you’ve published your research.

[Slide titled “Application”]

Gesche Huebner: Pre-registrations are more common in quantitative theory testing work. But they can be applied in any type of study including qualitative studies and also modelling studies.

[Slide titled “Every pre-registration should:”

• Specify the aims of your research
• Explain how you will access or create your data
• Outline what your approach to data analysis is]

Gesche Huebner: What you do in any type of pre-registration is you specify the aims of your research, how you will go about creating or accessing data, and what your approach to data analysis is.

[Slide titled “Quantitative studies should also include:”

• Your key outcome variables
• Your statistical analysis
• Any concrete hypotheses you are testing in your studies]

Gesche Huebner: For quantitative studies you normally also specify your key outcome variables, your statistical analysis, including how you will deal with outliers and missing data, and any concrete hypotheses you are testing in your studies. Templates exist for any type of research. For example qualitative work, quantitative studies, and also secondary data analysis.

[Slide titled “Why pre-register your study?”]

Gesche Huebner: So why might you want to pre-register your study? One of the biggest benefits is that it adds credibility to your results.

[Slide with an animated icon depicting a paper document with a pencil next to it and a tick appearing over the document and title “Pre-registration adds credibility to your results”.]

Gesche Huebner: If you pre-register your study, so you set in advance how you will analyse your data and have a timestamped version to show this, no one can accuse you of using questionable research practices.

[Slide titled “Keep your study safe from accusations of:”

• Changing your hypotheses
• Deleting outliers to get significant results
• Omitting certain results from your findings]

Gesche Huebner: So you cannot be accused of, for example, changing hypotheses afterwards, or of deleting outliers in a certain way to get significant results or just omitting certain results from your findings. Secondly, pre-registration helps to overcome the file-drawer problem and mitigate publication bias.

[Slide with an animated icon depicting a globe over the top of a filing cabinet and title “Pre-registration helps overcome the ‘file-drawer problem’ and mitigates publication bias”.]

Gesche Huebner: We, as in we the academic community, think that journals favour significant, novel, exciting findings. And in fact it’s known that journals do tend to rather publish those studies than null effects.

[Slide titled “The ‘file-drawer problem’ is an issue because:”

• Researchers repeat studies that have been shown not to work
• It wastes time and resources
• It gives existing findings greater credibility]

Gesche Huebner: However, this reduces the efficiency of science because others might try to run the same study again that has already been shown not to work. And hence we just waste time and resources on this. Also not publishing null results means that existing findings are given greater credibility because anything that might go against them and might not replicate them is not being published. Of course, before you design a new study, you won’t go through every pre-registration out there in the web to see what has been done already. However, a systematic review might just do that. So, pre-registration offers a transparent catalogue of all the kind of research that has been planned and then has been done. Some journals now even allow a publication form that’s called registered reports.

[Definition slide with title “Registered Reports: the submission of a very detailed pre-registration plan in order for a journal to provisionally commit to accepting your paper after you have got the results”]

Gesche Huebner: In those ones you submit a very, very detailed pre-registration plan and say exactly what you will be doing, and the journal provisionally commits to accepting your paper after you’ve got your results without knowing what those actually look like. Finally, pre-registration has direct benefits to you, the researcher.

[Slide with an animated icon depicting a magnifying glass being held over the top of a computer screen that allows data to be seen more closely and title “Pre-registration helps you think through your study and spot any mistakes in advance of data collection”.]

Gesche Huebner: It really helps you to think through your study before you start collecting the data, so you can spot any mistakes before you spend the time and effort to collect the data. It also helps to front load a lot of the work. So because you plan your analysis before you actually do it, once you have the data it’s very easy to just run the analysis. For example, I was recently doing a study using secondary data and the data set had hundreds of variables that I could have used as a predictor in my regression analysis. But because I planned my analysis, because I looked from a theoretical point of view which variables would be expected to matter, I was able to plan this well in advance. And then when I had the data and I ran my analysis, I knew exactly what to do. So instead of spending lots of time building lots of different models, I just knew which model to build and run and look at the results.

[Slide titled “Don’t worry if you pre-specify a study but something doesn’t quite work when you come to the analysis”]

Gesche Huebner: Sometimes it happens that you pre-specify a study, but something doesn’t quite work when it comes to analysis. Maybe the data are distributed differently than you would have expected. Maybe you have fewer cases of a certain characteristic, so you need to change something. And that’s perfectly fine, as long as you note in any publications how and why you deviated from your pre-specification.

[Slide titled “Potential Concerns”]

Gesche Huebner: One concern that sometimes arises around pre-registration is how it fits with more exploratory types of analysis. And that’s perfectly fine. So you can, in the same study you can pre-register some parts and then also conduct some additional exploratory research. As long as you specify in your manuscript which parts have been pre-registered, and which parts have been purely exploratory that’s perfectly fine. Sometimes we do a study where we do purely exploratory work. So for example we might fit different models for the same data to see which model fits best. So in this case your pre-registration might focus on the aims of the study, how you will access and clean the data, and which metrics you will be using to decide which model is the best. So whilst you might not specify exact analysis of your data, all the other benefits of pre-registration still apply.

[Slide titled “Practical Tips”]

Gesche Huebner: Let’s finish off with some practical tips on how to do pre-registration. So the very latest you would pre-register your study is before you access the data. Though ideally you would do it much earlier in the design stage so you can plan the study in the best possible way. So when you want to do a pre-registration the first thing that you do is to identify a suitable template.

[Slide containing a screen recording of the Open Science Network ‘wiki’ page on a plain turquoise background. The screen recording zooms in to show the different templates available in more detail.]

Gesche Huebner: For example, the Open Science Framework web page lists various templates.

[Slide containing a screen recording of the Open Science Network ‘new registration’ page. The screen recording zooms in and out and scrolls down the webpage to show the different information you would need to input to fill out the new registration form.]

Gesche Huebner: Then you open this document and you would be first asked to specify a title, the contributors, the research aims. And then depending on the type of study, for example, the hypotheses you will be testing, statistical analysis planned, how you will deal with missing data, how you will be dealing with outliers. And for example, how you code up categorical variables. Once you’ve filled in all the details, you can either press the submit button and then this will be sent to a registry and given an identifier so that others can look at this. Or sometimes you also just have an offline document that you then upload online to a registry. There are plenty of resources available online that give you further information about pre-registration. We have put together a table that gives you an overview of the most common templates to use for pre-registration.

[Slide titled “Summary”]

Gesche Huebner: To sum up, pre-registration of studies means to specify in advance how data will be treated and analysed before you collect or access the data. This has multiple benefits.

[Bullet point slide:

• Adds credibility to your findings
• Helps you spot any issues with the study design in advance
• Helps mitigate bias in academic publishing]

Gesche Huebner: So for example it adds credibility to your findings, it also helps us spot any issues with the study design before you actually conduct the study. And then finally it helps mitigate bias in academic publishing.

[Closing slide 1: title “Up Next: Reporting Guidelines”]

[Closing slide 2: title “TReQ: improving the transparency, reproducibility and quality of your research”]

[Closing slide 3: title “For links to further resources and more about us visit: bit.ly/TReQtools”]

[Closing slide 4: UCL logo, CREDS: Centre for Research into Energy Demand Solutions logo with title “Supported by”]