Ancient China was characterized by a feudalism strikingly similar to what we find in medieval Europe. In both places, feudalism was used as a form of government based on a fragmentation and privatization of political power. Well, upon closer inspection China was not feudal after all. But then, neither was Europe. Feudalism had to be discarded entirely as a descriptive category. Or maybe that was too fast: in descriptions of medieval Europe, at least, feudalism can still be used “when carefully defined”.
Confused? This is merely one of the more spectacular examples of how historians have changed their interpretations over time, and how they disagree. The same happens on an everyday basis with other historical issues: just like in social science, new historical research is often published because it overturns or at least questions the received wisdom.
The fundamental problem of historical inference
More than thirty years ago, John Goldthorpe defined historical facts as “inferences from the relics” and pointed out that “from any set of relics, the inferences that can be made are infinite”. This is perhaps best illustrated by inferences made from archeological findings, which are very dynamic and flexible. Archeologists constantly turn up new relics, many of these discoveries leave a very wide room for interpretation, and new finds will often alter the interpretation of older finds.
This puts social scientists enlisting the work of historians and archeologists as empirical evidence in a difficult position – whether for the purpose of coding large-N datasets or to qualitatively compare cases or do historical narratives or process-tracing. The problem they face can be likened to the fundamental problem of causal inference. The fundamental problem of historical inference is that we do not really know what happened, normally cannot know, because we cannot directly observe the past just like we cannot directly observe causal links.
Indeed, when dealing with times and places where historians and archeologists are gatekeepers, we are one step further removed from the truth because assessing the sources require expert skills such as knowledge of dead languages and scripts. We are the captives of professional historians’ interpretation of “relics that are manifestly incomplete, almost certainly unrepresentative, and in various other ways problematic”.
How do we know that what a particular historian or even a wider community of historians and archeologists says is true is in fact true, or at least that it would pass muster among those better acquainted with the relevant historiography? And which historical interpretation should we believe and make part of the evidentiary record and which should we question and disregard?
‘Pick-and-mix’ in history’s sweetshop
In sum, social scientists doing history face a problem of uncertainty and a selection problem, wrapped into one. In 1996, Ian Lustick discussed particularly the selection problem in a stimulating and valuable article which to this day is often cited. But it seems fair to say that these issues have not played a very prominent role in the proliferation of historical research that has taken place since around the turn of the millennium. Lustick’s work is often referenced but mostly the idea seems to be to recognize that the issue exists. This even goes for textbooks. For instance, in his otherwise excellent book on Comparative-Historical Methods, Matthew Lange mentions the selection issue in passing, with the obligatory reference to Lustick, but does not elaborate on it.
This is paradoxical considering that the “problem of history”, as I have termed it in a recent article, is almost certainly aggravated in historical work by social scientists. Whereas a main goal of historians is to do valid descriptive inference, even to recognize the bewildering complexity of history, social scientists normally use history to generalize about causal relationships that have some sort of present-day relevance or at least a more general relevance for understanding human societies. As a profession, we are lumpers rather than splitters, we have a tendency to read history backward, and we face a much higher risk of confirmation bias in our reading of the historical evidence because we normally focus on testing a particular theoretical argument rather than trying to understand the historical context.
In 1991, Goldthorpe observed that the historical sociology that had proliferated since Barrington Moore’s seminal work “is not significantly rule-governed; its practitioners enjoy a delightful freedom to play ‘pick-and-mix’ in history’s sweetshop”. Today, the situation is clearly much better, both due to Ian Lustick’s work and because of an increasing professionalization of social science inquiry in general. That said, it still seems fair to say that many social scientists lift observations out of history in a way that is not significantly rule-governed.
Rules of engagement
In a four recent articles, I have proposed some simple rules of engagement that will at least make it more difficult “to play ‘pick-and-mix’ in history’s sweetshop”. The idea is to figuratively tie ourselves to the mast when putting together our evidentiary record – what Svend-Erik Skaaning and I have termed “The Ulysses Principle”.
Taken together, what I propose amounts to a change in cognitive style, as Albert Hirschman would say. We need to treat the prior historical work that we enlist for evidence in a critical way and clearly convey some kind of – qualitative or quantitative – uncertainty estimate in the way we present our findings. As Lustick has emphasized, this requires identifying the different viewpoints in the historical literature evidence is enlisted from. When there is no clear consensus (which is the rule rather than the exception), we need to explain why a certain interpretation is preferred, preferably in the main text but if there is no room in online appendices. Five more specific rules can guide this selection process and hence be seen as ways of diminishing the risk of importing bias (for illustrations, see the four articles):
- First, base evidence claims about historical facts mainly on work by professional historians (or archeologists) rather than by social scientists. This is to avoid importing selection bias into the evidentiary record by steering clear of evidence that is strongly colored by theoretical claims.
- Second, base evidence claims about historical facts mainly on relatively new rather than relatively old historical work. This is to avoid enlisting outdated interpretations as evidence, considering that new historical evidence constantly alters what we think we know.
- Third, ensure and document that there is a high degree of conceptual consistency between the work that is enlisted for evidence and your own definitions. This is important because the same concepts are often defined in strikingly different ways by historians and social scientists, or for that matter by different historians and different social scientists.
- Fourth, if possible, use specific concepts that are relatively easy to operationalize rather than composite and loose concepts. This serves, for instance, to diminish the risk of making false historical analogies that ride roughshod over differences in context between places removed in space and/or time.
- Fifth, always use page references for specific claims about historical facts so that other scholars can fact-check whether they agree with the interpretation of the source.
Needless to say, there will often be trade-offs between the guidelines (for instance, older work might be more in line with one’s own conceptual definitions) and in a specific research process one has prioritize those that seem most relevant and meaningful.
Simpler versions of these guidelines can also be used to review historical work by social scientists by flagging cases where the way the evidentiary record has been put together seems superficial, for instance because the empirical “evidence” is mainly based on references to interpretations made by social scientists who have already preselected evidence to make the case for some general theoretical argument or on surprisingly old historical material.
Ideally, we also need guidelines that shield social scientists against the temptation to read history backward. Here it is more difficult to establish hard-and-fast rules, though for instance the use of specific concepts should mitigate this problem. I will do no more than suggest two tentative guidelines here: First, avoid using anachronistic definitions that have been developed to understand later periods – especially contemporary societies – when venturing back in time. At the very least, discuss whether definitions developed to describe later periods (e.g., class, state, institutions of constraints) fit when venturing back in time. Second, discuss the possibility that factors which explain the origins of an institutions are not the same as those explaining its later consolidation – a case strongly made by Deborah Boucoyannis in a recent book. More generally, try to read what historians have written about the subject matter of interest in an open way, instead of only venturing into historiography to sort the historical evidence into preconceived general concepts used to apply strong theoretical arguments.
A call for epistemological modesty
Adhering to these rules of engagement is easier said than done. Looking back, much of my own historical work has been blemished by the problems of reading history backward and confirmation bias, and especially when I began working on historical subjects, I would often reference narrative observations from old work or from other social scientists.
A positive spin on this would be to say that I have developed the aforementioned guidelines through my own trial-and-error prone work. More worryingly, I still find it difficult to do as I preach. In particular, I struggle against the lure of writing up results without making the proper reservations – or in other words without recognizing the inherent uncertainty of results that rest on biased and incomplete historical data.
It might be that the suggested straitjacket is, on balance, more of a hindrance than a help and that this is the reason that it is so difficult to consistently do. But there is a more disturbing possibility, which is that the problem lies with the incentives. As a profession, we are not rewarded by pointing out how uncertain the historical underpinnings of our results are. Rather, we are rewarded for presenting clear findings, in a brief and concise text that rather signals certainty than the opposite. And the problem lies not only with the wider audience (including reviewers and editors) but with ourselves: We are trained to tell a clear story rather than one which is tentative and uncertain. As historically minded social scientists, I think we have a duty to alter this stage of affairs, depending of course on the quality of our evidence, even if we can never dispense with some degree of confirmation bias. “[A] bit of epistemological modesty is in order” as a recent paper by Charnysh, Gehlbach, and Finkel taking stock of the field of HPE puts it.