Guess Who’s Back; PAP Again
Awhile ago, I wrote a Broadstreet post on the use of PAPs with HPE. Since then, I’ve thought a bit more about this, as part of a chapter I’m working on for my book on historical data. Preregistration is a foundation of experimental work, but is also being suggested for observational and qualitative research as well — I think it’s important that HPE engage with this movement.
When I workshop this chapter, I feel like the first reaction from the audience is “No way, my historical data is messy, my process is inductive, there’s no way I could preregister anything.” But ultimately, the claim that “historical data is messy” should not be an excuse for not doing a PAP — it should be a reason for doing it!
TLDR; I think there IS utility in pre-analysis plans for quantitative historical research, though it depends much more on the project.
As a general scope condition, it’s worth noting that PAPs are not designed for purely exploratory research. So any use of a PAP with historical data is best paired with confirmatory analyses that tests empirical hypotheses. But note a PAP doesn’t prevent a researcher from conducting unregistered analyses or engaging in exploration, but just allows researchers to delineate to readers in the final paper what was testing versus exploration (Humphreys, Sanchez de la Sierra, and van der Windt 2017).
I think PAPs are a great way to structure and document the research process, which can help make the overall project more rigorous (and knowing the types of analysis you want to run and variables you need can help plan data collection, saving costly trips to the archives!). But I’m trying to think more clearly about when PAPs would help mitigate publication bias, and be a credible signal.
I think the point made by Jacobs (2020) is very useful — PAPs are not binary in terms of their effectiveness, instead, it’s a continuum. Some PAPs have more credibility, and can provide more research transparency, than others (aka the extent to which the researcher can credibly commit to not having seen and manipulated the data). There are conditions under which PAPs offer more transparency for quantitative research using historical data. Research studies, or cases, can be strong or weak; the strongest cases for credibility come from situations where the data is inaccessible, or has not yet been collected; the weakest cases come from where the data is already online and downloadable.
There are four general cases that might come up, when working with historical data:
Case 1: Yet-to-be-released Historical Data: The strongest case for a historical PAP comes from situations where the data has yet to be released. While this seemingly wouldn’t apply to historical data, there are cases where this happens. National census are an example — the United States delays population census releases, for anonymity purposes, using what is called the “72-Year Rule.” Records are only released to the general public 72 years after census day; the 1950 census records were recently released in April 2022.
Case 2: Full Collection and Digitization of New Archival Data: A moderately strong case for a historical PAP can be made for data that a researcher collects and digitizes; in other words, historical data that is not in analyzable form. Particularly if this data exists in restricted or generally hard to access archives, though observational, data that is newly collected and therefore newly accessible to the researcher (and the field at large) could warrant a PAP.
Case 3: Partial Collection and Digitization of New Archival Data: A related case is the partial collection and/or digitization of new data, designed to be added to a preexisting dataset. Here, the historical data could be the explanatory or the outcome variables; historical data could also be paired with modern data. While there is no way for the researcher to credibly signal that they didn’t explore the already public historical dataset, a PAP registered before the additional data is collected aims to signal credibility on the basis of the new data.
Case 4: Data Available Online: The weakest case for PAPs comes from a study that only uses historical data that is publicly accessible. Historical data can be found in public repositories (such as ICPSR, Harvard Dataverse) or as replication materials from already published work. A PAP, if conducted would do little to guarantee that preliminary analysis wasn’t conducted in advance. The researcher still may benefit from the effort and rigor in research design stemming from a PAP, but it can’t mitigate publication bias in this case.
Not every observational study with historical data can be credibly preregistered, and there are many cases where the signal is weak and the exercise is complex such that it may not productively contribute to advancing the research. But given that PAPs aren’t binding, in that they are not required for observational data, any PAPs sincerely undertaken could potentially strengthen historical research.
I’ll end on an open-ended question — it’s clear that research transparency, and formalizing our thinking about the research process would be useful in historical data. But is pre-analysis plan the correct format? Or rather, what additional steps do we need to document our process, and should we fit it into the existing mold of a PAP? It might be possible to include extensive documentation about data collection, interpretation, and contextual learning elsewhere — on the scholar’s website, in online appendices, or maybe in new templates. Yet there are gains from all using a consistent format. This is an open question, stay tuned!