Those disciplines and fields that understand themselves as primarily humanistic have long had a special interest in exploring nuance, uncertainty, and ambiguity in our objects. While data-driven research might frequently be assumed to be in opposition to such concepts, this is not the case. Data may, in fact, enable the discovery and foregrounding of ambiguity. Yet as computational methods, especially those designed to operate on large-scale textual archives, increasingly enter into the work of such humanists there is a risk that access to those ambiguous dimensions of data will be lost. This risk is as much present for the researcher using these tools to interpret their objects as it is for the skeptical reader or reviewer of such interpretations. Visually appealing graphic renderings of data, high classification accuracy and confidence scores, and impressive summary statistics can easily be rhetorically structured to appeal to preconceived notions and commonly accepted understandings. Effective and academically responsible argumentation and interpretation in the humanities requires not just openness, which is to say unfettered access to the methods and data used to generate visualizations and to demonstrate accuracy, but that the selected datasets and models are interpretable. When features are derived from text, as in the case of much of the machine learning taking place in the humanities, these features need to be exposed along with the data that helps readers understand their significance. Humanistic work that is primarily argumentative — it should be acknowledged that of course not all digital humanities work is argumentative — depends upon certain frameworks by which readers can evaluate claims. These frameworks generally operate within established interpretive communities of scholars working on similar problems or using similar approaches [Fish 1980]. While data and statistics have had a minimal presence within the humanities prior to the twenty-first century, contemporary computational approaches, if presented properly, are highly compatible with existing humanistic argumentative frameworks [Dobson 2019]. While some computational fields have been undergoing a reproducibility crisis, the present discussion is not primarily an argument about access, an argument that would critique the use of so-called black box algorithms or advocate for the digital humanities to join in the movement toward open science, but rather this essay argues for the importance of listening to rather than hiding noise, for exposing complexities and ambiguities, and for interpreting the nuances within data derived from our objects. It will do so by explicating the role of hermeneutics in the digital humanities before turning to three critical sites within computational workflows that will serve as case studies: the creation of data objects, topic modeling methods, and classification algorithms. These three sites, with their increasing complexity, enable us to see key common limits to interpretability across the software “stack” of tools commonly used in the digital humanities today.

Joining the recent injunction “against cleaning” made by Katie Rawson and Trevor Muñoz, who argue that data cleaning, standardization, and “tidiness [privilege] the structure of a container, rather than the data inside it” [Rawson and Muñoz 2019, 290], this essay turns to basic computational objects used in text-based machine learning, data structures, model formats, and classification algorithms, as a case study for the preservation of nuance, diversity, and interpretability. Similar to the way in which Rawson and Muñoz call data cleaning a “diversity-hiding trick,” selecting certain models and data formats as containers can suppress evidence and minimize the presence of the unexpected, the minor, and the troubling, leading to a misrepresentation of the contents and unwarranted claims. The following critique also shares the ethical force of the request for researchers to establish and follow clear documentation procedures for producing and using machine learning datasets and models made by Margaret Mitchell, Timnit Gebru, and their co-authors [Mitchell et al. 2019] [Gebru et al. 2018]. The combination of prioritizing methodological transparency and foregrounding the complexity and diversity of data is necessary in order to preserve scholarly standards and to enable, within the computational environment, the dialogical forms of argumentation common to the humanities.

Data, if we construe data broadly to mean evidence presented to readers or listeners in support of an argument, has long existed within the argumentative apparatus used within humanistic fields. Forms of data include textual evidence put forward in a close reading or material culture gathered from an archive. Evidence is made explicit by authors in academic arguments. Evidence can be variously interpreted by readers and observers. It is another feature of the academic argumentation that frames and focalizes evidence. Warrants, to borrow a term from the philosophy of logic — in particular the work of Stephen Toulmin, function within argumentation as the implicit link between evidence and claims [Toulmin 1969]. Strong arguments involve making explicit that which implicitly provides the foundation for the interpretive claim, stating the warrants alongside the arguments that leverage them. I invoke the language of formal argumentation in order to help frame some emerging problems within computational work within the digital humanities.

But we first need to disentangle explanation from interpretation. Recently, there has been a growing demand for explainable models within artificial intelligence research and especially within those social scientific fields making use of AI techniques. These efforts have primarily asserted that explanation and interpretation are the same [Miller 2019]. Explanation, however, does not have the same meaning as interpretation because an explanation can still function without resolving uncertainty and ambiguity found within data while an interpretation can and frequently does take these up as the basis for exploration or argumentation [Burckhardt 1968]. Interpretation arises from an encounter with evidence. In the humanities, interpretation tends to foreground uncertainty and ambiguity within this encounter; the interpreter may locate these sites within the primary object or gesture toward other evidence. Interpretation thus needs to be in relation to presented evidence, whether this evidence is an image, a passage of text, or information derived from data.

Computational models tends to reduce uncertainty by only registering that which is classifiable according to the logical structuring of the model as a container. While Julia Flanders and Fotis Jannidis are concerned with text-markup models and frameworks for complex reference systems, their assertion that “most modeling systems at their core require uncertainty to be represented in the content (i.e. through qualifying notations) rather than in the modeling itself” applies equally to numerical models of textual sources and the typical ways in which these are used in the application of machine learning in the humanities [Flanders and Jannidis, 21]. Data-driven arguments in the humanities need to present interpretable outputs because in order to accept arguments as plausible, readers need to be able to understand if the claims are warranted by evidence. The rhetorical situation within existing communities of interpretation is structured according to the logics of plausibility and therefore interpretive arguments cannot only be evaluated according to the governing logic of the correct and rigorous interpretation of data. From the basis of a humanistic approach toward tools and data, interpretable outputs from computational models derived from textual sources are simultaneously qualitative and quantitative. This is to say that plausibility and the qualitative are frequently registered in quantitative features and attributes including edge cases, outliers, anomalies, ill-fitting values as well as cases of noise taken as signal and vice versa.

There are two major categories in which we might place most computational humanities work at present. The first category is more affiliated with the practices of literary criticism and it draws upon the long history of understanding criticism not as secondary to the creative work that it interprets but itself as a creative act [Hartman 1980, 189–213]. We can understand this mode of criticism as involving a collaborative relation between critic and reader that is playful yet built on shared norms. Creative computational criticism does not explicitly depend upon scientific rigor associated with statistics and computation such as validity, falsifiability, and reproducibility. Rather, algorithmic transformations of text are framed as creative or even speculative renderings. Criticism in this vein can be argumentative but the grounds for the critical work are not subject to a mode of critique that would prove it wrong, obsolete, or invalid. Such forms of computation are primary playful; these readings transform and deform input text. This form of computational work might best be associated with the work of Stephen Ramsay and Nick Montfort, although many others frame their work in similar terms [Ramsay 2014] [Montfort 2018]. The second category involves the use of computation for research in order to make empirical claims about data. These practices and methods share a desire to demonstrate the significance of any findings with the social sciences. Researchers working in this area thus produce arguments that require the methods to be statistically sound. Interpretations of their results are warranted by a series of assumptions that can be tested and verified. These scholars make hermeneutical claims about computational models that are grounded in their analysis of data. While they might make use of similar algorithms as those that I have classed as creative computational critics, they are less interested in engaging deformations than refining models to produce more significant results.

Though it provides some level of access, the explicit presentation of data in support of an argument, a discovery, or a finding does not necessarily mean greater interpretability. One increasingly common example, machine learning classification data, are frequently presented in the form of accuracy scores and the confusion matrix. Parsing f-scores, and precision and recall values enables a certain degree of understanding of the overall performance of the classifier and the function of the workflow as a whole, but this leaves aside many other post-classification metrics that can be directly applied to the underlying data model. These data can tell us much about our input datasets and the criteria by which the classifier made its classifications. For applications in the humanities, in which there is a scholarly community that cares deeply about ambiguity, nuance, and the historicity of language, models that can provide extended interpretable output features, for example coefficients, probability values, and weights used in classification are necessary. The exposure of these extended interpretable features enables some shifting of scale — a shifting that matches the needs of humanistic interpretation and recognizes that while meaning can be made at the level of individual signifier, the word or token and its frequencies, it is in those larger units, for example the sentence, paragraph, chapter, and volume, that much of the activity of meaning making is to be found [Algee-Hewitt et al. 2017][Allison et al. 2017]. Just as decontextualized individual signifiers can all too easily hide ambiguous meanings and data diversity, the absence of interpretable features frequently leaves readers of computational work with many unknowns as lists are leveled, distributions smoothed, and complex geometries collapsed.

Tool criticism is an emergent category of analysis that provides a humanities-centric framework for understanding the use of computer-aided methods in both the humanities and the sciences. Tool criticism seeks to foreground the tools used in humanities research that make use of computation. In so doing, it foregrounds methodology for computational researchers and for their critics. Karin van Es, who has done a great deal to develop and mobilize this concept, asks researchers to understand the epistemic impact of their tools. Tool criticism framework requires access to tools and data and produces inquiry into the affordances of computational methods:

Tool criticism is the critical inquiry of knowledge technologies considered or used for various purposes. It reviews the qualities of the tool in light of, for instance, research activities and reflects on how the tool (e.g. its data source, working mechanisms, anticipated use, interface, and embedded assumptions) affects the user, research process and output.  [van Es et al. 2018, 26]

Understanding computational methods through the language of tool criticism helps foreground the need for a robust inspection of input data, the workflow, and algorithmically generated output data. In many ways, the preference for interpretable data is the output version of input solutions offered by scholars interested in better descriptions of input data. In Katherine Bode’s notion of the data-rich object, we find a strong argument for better descriptions of input data, especially when these data concern book history. The data-rich object, in Bode’s account, provides a conceptual framework through which we might bridge the gap between scholarly work in bibliographical studies and computational text analysis in literary studies [Bode 2017]. While some shared repositories provide a set of reference texts and features extracted from these texts that have been designed for computational work (see the HathiTrust Extracted Features Dataset for an example of this type of repository) once these input objects enter into scholarly workflow, much information has been stripped from the input data objects making it much more difficult to do cross-study comparisons and to apply these insights to ongoing scholarly debates within literary critical communities [Capitanu et al. 2016].

In many computational fields it is a norm that researchers, especially academic researchers undergoing peer review and working with complex technological instruments, make their methods transparent and include their data, models, as well as the code necessary to implement their research. Academic researchers are making demands of each other that they open up the black box of their experimental apparatus, much in the way that critics and activists have made similar demands of corporate and government entities deploying proprietary and secretive algorithmically-driven decision-making tools. The “open science” movement champions such a model of methodological transparency as a solution and remedy for what has come to be recognized as a replication crisis in several fields, most notably in the psychological and brain sciences. Open access and transparency into methodology, however, do not ensure that reviewers and others will be able to understand the data-driven claims and the argumentative grounds supporting such claims. Mike Ananny and Kate Crawford critique what they call the “transparency ideal” for its emphasis on openness as the single solution to accountability in technological systems (Ananny and Crawford 2018). Ananny and Crawford understand accountability in terms not just of the operation of a single element but rather through the social critique of systems and in particular the workings of power throughout the complex social systems in which algorithms and other computer-based technological tools are embedded. Johannes Paßmann and Asher Boersma argue in “Unknowing Algorithms: On Transparency of Unopenable Black Boxes” for another mode of interpretation applicable to computational models that might be considered supplementary or an alternative practice to the demands for openness driven by the transparency ideal, a practice “that is not so much concerned with positive knowledge, but that deals with skills which help dealing with those parts of an artefact that one still does not know” [Paßmann 2017, 140]. They cite situations like medical procedures in which an actor — an actor who importantly is not the primary actor but perhaps another physician — might gather unspecified and potentially useless data as a form of an unopenable black box; the collection of data and experience cannot be explained or defended in methodological terms but it might provide useful information for later procedures. This example provides them with a way of describing negative knowledge, an understanding of the known unknown. Paßmann and Boersma use such a scene to defend a notion of transparency that is distant yet still rooted in phenomenological experience. They offer, via Maurice Merleau-Ponty’s account of the way in which a woman wearing a hat equipped with an extended feather navigates space, a tool-based mode of exploratory analysis, “a carefully paced out unknowing” that can limn the space of known unknown [Paßmann 2017, 145]. This amounts to what they call “feather knowledge,” an alternative to direct contact that might be capable of providing knowledge about the workings of computational models when such direct access is not available or not possible.

Paßmann and Boersma’s concept of distance and mediated access to black boxes should not be taken as an endorsement of a refusal of access but an interpretive strategy that enables researchers to probe and feel their way around propriety technology. There are situations within academic research in which some models might need to be protected, for example if they might leak identifiable training data about human subjects, but in the vast majority of cases, open access is essential to understand the meaning of the data and subsequent data-based claims. At the same time, feather knowledge might contribute to the understanding of computational models in ways that are fundamentally different and perhaps even more crucial than disembodied observation of models.

Even if it were possible to achieve the goals of the transparency ideal, certain methods require other forms of inspection than observation of function in order to be understood and explainable. Direct access to parameters and features supplied as training data for classification algorithms can enable queries to examine the distribution of certain words or phrases in order to understand if these are indeed representative of the modeled event or phenomenon. For example, counterfactual testing of a model and its output can be done simply by using the same model with other samples, with data known to the counterfactual investigator. While counterfactual testing fits into the general design of a tool or workflow, sometimes this might not provide enough useful information to evaluate the model and to expose the diversity of data within the model. Developing an explanation of how some computational functions work can involve treating the algorithm or model as something to be tricked through adversarial techniques. Adversarial techniques might involve observing reactions to unexpected input, for example, and rather than supplying expected textual features one might supply randomized data or even alternate numerical input. This method of probing the functioning of an algorithm can leak or reveal decision criteria or training features if these are not available. Needing to follow such procedures might be considered undesirable due to the unreliability of such methods of probing. By making use of highly interpretable data and algorithms, the digital humanities community can avoid needing to turn to adversarial techniques to expose nuanced and diverse data and aid in the verification and interpretation of computational models.