Increasingly, research policy-makers and funders are calling for inter- and transdisciplinary approaches to be applied in response to complex socio-scientific questions. Whilst the most exciting and ground-breaking innovations, to paraphrase Carlos Moedas, former European Commissioner for Research, Science and Innovation, may indeed be happening at the intersection of disciplines, this kind of cross-disciplinary collaboration poses many challenges, among which are the individual disciplinary values, concepts of proof and evidence, languages, technical cultures and methodologies employed by different domains ([Bruce et al 2004]; [Lowe, Whitman, and Phillipson 2009]; [Edmond et al. 2018]). It is perhaps unsurprising that overcoming the challenges associated with specialised terminology/vocabulary is often instrumental to the success of cooperative work involving individuals of different disciplinary backgrounds and expertise levels ([Skeels et al. 2008]; [Bruce et al 2004]). This may be, in particular, the case when the goal of the cooperation is to build a software platform, within and behind which mutual misunderstandings can become encoded, creating dissonances in structure and function, and potentially leading to difficulties for future users. This paper contributes to the academic literature on how a specific kind of technical artifact – in this case, a taxonomy of uncertainty – was used to enhance the functionality of a technical team and their co-developed platform itself created for the purpose of supporting interdisciplinary cooperative work. More specifically, we discuss how the cross-disciplinary design of this taxonomy became not just an instrument to organise data, but also a tool to negotiate and build compromises between four different communities of practice, each of which came to the project with a very different epistemically ‘native’ approach to issues of uncertainty. In order to illustrate the many interlinked functions, the taxonomy was required to perform, we will briefly introduce the project, its aims, and the four diverse but complementary partners that came together to deliver it. We will then discuss the taxonomy itself, how it needed to be adapted from earlier, similar work, and some of the issues it allowed the project team to negotiate and discuss. In the final section, we will look at the application of the taxonomy and how it informs system function in a manner that is sensitive to the requirements of all of the disciplinary stakeholders in the project.

The project in question is the PROgressive VIsual DEcision-Making in Digital Humanities (PROVIDEDH) project, which aimed to develop a web-based multimodal collaborative platform for the visual analysis of uncertainty in the Digital Humanities (DH) data analysis pipeline (discussed in further detail below). The consortium can be described as both complex and nonroutine [Strauss 1988], comprising a multinational team of four institutional partners and two overlapping sub-teams (composed of the two technical partners and two partners representing the arts and humanities). Further, the individual partner teams combine researchers and engineers of varying levels of expertise in specific branches of computer science and humanities disciplines as well as in open innovation and DH.

The PROVIDEDH project was originally conceptualised according to the imperatives of computer science as a discipline. The coordinator, based in the Visual Analytics and Information Visualisation Research Group (VisUsal) at the University of Salamanca (USAL), brought expertise in visual analytics, research methods, software engineering, and human-computer interaction (HCI). In addition to the project coordination activities, USAL led both the development of user interfaces and evaluation and the research for the definition and management of uncertainty in DH. The second technical partner, the Poznan Supercomputing and Networking Center (PSNC), brought extensive experience designing and building large-scale DH infrastructures, and led the development of the PROVIDEDH platform. Considering its intended user base, however, the technological expertise of USAL and PSNC alone would not have been sufficient to realise the PROVIDEDH platform. Realising the principle “that [Digital Humanities] tool builders must consider themselves as entering into a social contract with tool users” [Gibbs and Owens 2012], two further partners, Trinity College Dublin and the Austrian Academy of Science (in conjunction with the Ars Electronica Research Institute Knowledge for Humanity), were brought on to contribute from the perspective of Digital Humanities and Open Innovation Science respectively, and led research regarding user needs, user engagement, and exploitation and dissemination of results.

One of PROVIDEDH’s first objectives was to define and describe the key types of uncertainty in DH research datasets. To date, various definitions, classifications and taxonomies of uncertainty have been proposed, but much of this work on the characterisation of uncertainty has been delivered within isolated domains. As Kouw et al. (2012) argue, typologies of this kind are valuable tools in the attempt to determine the extent to which uncertainties can in fact be explained. Perhaps more importantly from the perspective of PROVIDEDH is the authors’ assertion that new forms of knowledge production that challenge our understanding of uncertainty “almost always occur at the boundaries between different disciplines.”

As described above, the PROVIDEDH team is diverse with goals that could only be met via closely aligned collaboration. A recent report by the EU-funded SHAPE-ID project, which addressed the challenge of improving inter- and transdisciplinary cooperation between Arts, Humanities and Social Sciences (AHSS) and Science, Technology, Engineering, Mathematics, and Medicine (STEMM) disciplines, identified 25 factors that influence interdisciplinary success or failure, depending on the circumstances of a project [Vienni Baptista et al. 2020]. One of the first and primary challenges faced by the PROVIDEDH team relates to what Vienni and Baptista (among others) describe as the “Communicative” factor, which refers to the different “languages” employed by different disciplines ([Lowe, Whitman, and Phillipson 2009]; [Boix Mansilla, Lamont, and Sato 2016]; [Vienni Baptista et al. 2020]). The absence of a common transdisciplinary language/terminology to discuss shared concepts such as uncertainty can substantially hinder interdisciplinary research and, conversely, processes to resolve this gap are exceptionally useful for illustrating how the development of shared intellectual artifacts can create a common ground for exchange.

Even among groups with apparently similar disciplinary backgrounds, conceptual frameworks may differ substantially ([Jeffrey 2003]; [Pennington 2008]). To cite one example, research conducted by the Knowledge Complexity (KPLEX) project found that inconsistent and contradictory statements in academic literature on such a foundational word as ‘data’ are manifold, even and indeed especially, within the field of computer science. In a series of interviews conducted with computer scientists, the project team found the trend “point[ing ]towards an epistemic cultural bias towards viewing data, whatever it is, as broadly encompassing, and in terms of its function or utility in the research project, rather than a complex set of information objects that come with biases built in to them, and which might merit a certain amount of meta-reflection” [Edmond et al. 2022]. In essence, the discourse of data in computer science appears indicative of what Edward Hall would call a ‘high context’ culture in which the precise context, in which a word is used, can determine its intended meaning in a way that might be opaque to outsiders. Needless to say, if the possibility for variant understandings can be so high within a single discipline, the gaps between the humanities (a much lower context culture) and computer science surely add to the difficulty of creating a unified classification of certain phenomena. The discourse, or more accurately discourses, on key terminology such as data or uncertainty can have a serious delimiting effect on the potential of cross-disciplinary, cooperative projects:

”[O]ur traditional reliance on community ties to overcome the flaws in both our data and the terminology we use to speak of it do not translate well to larger scale interdisciplinary endeavours, to environments where the backgrounds or motivations of researchers/participants are not necessarily known or trusted or to environments where either the foundations of the research objects (such as is found in big data) or those of the algorithmic processing results (such as found in many AI applications) are not superficially legible to a human researcher.” [Edmond et al. 2018, p. 10].

In this same report, the KPLEX project also set forth a number of recommendations for fostering fruitful interdisciplinary collaboration, including a commitment to the negotiation of key terminology and hierarchies [Edmond et al. 2018, p. 13]. As a first step then, a project must establish a baseline for collaboration. However, the requirements for this baseline, what can and cannot be taken for granted, are very much predicated on the starting position of those seeking to cooperate, which we outline briefly below.

In recent years, the graphical display of uncertainty has become an important trend in visualisation research ([Kay et al. 2016]; [Hullman et al. 2018]; [Kale et al. 2018]; [Hullman 2020]). For this, Digital Humanities represents an exciting field of experimentation [Benito-Santos and Therón Sánchez 2020]. In this regard, the first notable efforts to address the visual representation of uncertainty originated in the fields of geography and spatial data visualisation with proposals describing uncertainty taxonomies whose categories could be mapped to the different available visual channels ([MacEachren 1992]; [Fisher 1999]). For example, MacEachren’s employed the concept of “quality” to build his first uncertainty taxonomy, and accounted for the different manners in which uncertainty could be introduced into the data according to the stages of the typical data analysis pipeline. According to this, he described concepts such as accuracy – the exactness of data –, and precision – the degree of refinement with which a measurement is taken. Later in the text, MacEachren mapped these concepts to several visual variables (e.g., saturation), building on the famous previous work on semiology by the French cartographer Jacques Bertin [Bertin 1983]. In more recent work, MacEachren completed his previous work with two empirical studies focused on uncertainty visualisation in which a typology of uncertainty was matched against the spatial, temporal and attribute components of data [MacEachren 2012].

In a different approach, Fisher [Fisher 1999] proposed an uncertainty taxonomy elaborating the concepts of well- and poorly-defined objects, which are used to define the categories into which uncertainty can be classified. More concretely, Fisher’s taxonomy presents a particular point of view within the more verbose taxonomy of ignorance developed by [Smithson 1989]. According to Fisher’s views, uncertainty related to well-defined objects is caused by errors and is probabilistic in nature. This type of uncertainty is typically referred to as “aleatoric” or “irreducible”. On the contrary, poorly defined objects relate to the concept of epistemic uncertainty, which can in fact be reduced. More recently, Simon and his colleagues adapted Fisher’s taxonomy in their work on numerical assessment of risk and dependability [Simon, Weber, and Sallak 2018], which in turn served us a starting point to propose our own uncertainty taxonomy in the context of Digital Humanities research. Simon et al.’s adaptation of Fisher’s taxonomy is reproduced in Figure 2 below:

The original uncertainty taxonomy by Simon et al. includes four more categories into which epistemic uncertainty might be classified: imprecision, ignorance, credibility and incompleteness. According to the authors, “imprecision corresponds to the inability to express the true value because the absence of experimental values does not allow the definition of a probability distribution or because it is difficult to obtain the exact value of a measure.” The next category, ignorance (which can be partial or total), refers to “the inability to express knowledge on disjoint hypotheses.” Incompleteness makes reference to “the fact that not all situations are covered.” This is especially applicable to Digital Humanities research objects which may have been degraded or partially lost due to natural processes or the different transformations that may have been applied to the originals. Finally, credibility concerns “the weight that an agent can attach to its judgement,” [Jarlbrink and Snickars 2017] and it is also of special importance to the aims of our project. For example, a researcher may or may not trust the observations made by other agents who have previously worked on the same data, according to her degree of grounded knowledge on the subject and personal preference and/or bias.

Although a strong starting point, none of the existing taxonomies was able to satisfy all of the partners in PROVIDEDH, as the balance between refinement and actionability was, in each case, never mutually recognisable. As Skeels and his colleagues noted in 2008, uncertainty is referred to inconsistently both within and among domains, they continue that “[w]ithin the amorphous concept of uncertainty there are many types of uncertainty that may warrant different visualisation techniques.” [Skeels et al. 2008] The challenge is, therefore, not to arrive at a single definition that obscures differences of uncertainty, but to build dialogue between different research domains while recognising their differences, and ensuring that any definition of uncertainty the project as a whole would adopt could be applied and implemented by all partners in their own research contexts. This challenges the idea that it is possible to create a single view of uncertainty. “While any group’s ontology is unlikely to match that of every individual within the group, the extent of mismatch tends to increase with the scale of the group and the differences between the purpose of individual and group ontologies” [Wallack and Srinivasan 2009, p. 2]. The reality is that the development of a taxonomy, which must cater to the needs of a range of disciplines and stakeholders, may often lead to a result in which the specific requirements of not one subject area or stakeholder group are fully met, and yet which all can still use at some level of granularity.

Building on these insights, the group organised a series of shared workshops exploring the issue of conceptions of uncertainty and its relationship to decision-making in different contexts. These workshops commenced from existing taxonomies, such as Fisher’s (reproduced above), and sought to revise them so as to reach consensus, if not agreement, about how conceptions of uncertainty might be made concrete and actionable in a historical research system. The three workshops each approached a different aspect of the overall challenge, focusing first on the historian’s requirements for decision-making as compared to the system's needs for clarity and simplicity. The second continued this discussion with a wider and more diverse group of potential users, and the third looked at the same challenge from the perspective of open innovation. Robust discussions and negotiations about the taxonomy occurred at every one of these discussion points. For example, in the interaction designed to bridge what was perceived as a back-end processing ‘optimal’ with the perspective of the historian, nearly every term had to be renamed, glossed and refined, with, for example, the descriptor of ‘error’ being rejected as lacking sufficient precision. The domain experts also strongly advocated for the recognition of time-based element tracing the possible source of uncertainty from the original moment of a source’s creation through the many transformations over time (physical damage, cataloguing, transcribing, translating, digitisation, etc.) all the way through to the task of the researcher itself. Although later discussions continued to be active and important for the purpose of converging both understanding and usage patterns (in particular the meeting to discuss how to further extend the taxonomy’s use in the Open Innovation context), fewer changes were implemented over time.

Departing from the taxonomies proposed by [Fisher 1999] and Simon et al. [Simon, Weber, and Sallak 2018] that were introduced in the previous section, the project partners developed an HCI-inspired uncertainty taxonomy aimed at producing effective visualisations from humanities textual data and related annotations (see Figure 4). Findings in these workshops suggested that a matrix-based taxonomy would be of the most use to the project and the potential users of its outputs. This substantiated the earlier work conducted in the context of the KPLEX project (discussed above) which suggested that something like Peterson’s uncertainty matrix for simulation uncertainties in climate change [Petersen 2012] would provide the most effective approach for all sides. The key differentiator was the location or site of uncertainty. For example, Pang and his collaborators describe the different processes through which uncertainty can be introduced into data as: “the introduction of data uncertainty from models and measurements, derived uncertainty from transformation processes, and visualisation uncertainty from the visualisation process itself” [Pang, Wittenbrink, and Lodha 1997]. Petersen similarly distinguishes a number of locations of uncertainty: see his model below in Figure 3.

The matrix shown in Figure 4 was built as a result of the PROVIDEDH development process.

Although it maintains the overall simplicity of having four categories of uncertainty (about which much negotiation of terminology was required) the final taxonomy also incorporates a number of other dimensions deemed important for the notation of uncertainty. First, on the y-axis, it incorporates a sensitivity to time and the possible actors that might have introduced uncertainty into a system. This is particularly important for historical research, as an uncertainty introduced by a more recent process (such as dirty OCR of historical texts) may be much easier to manage in a decision-making process than an original gap in what was seen or reported. Similarly, the matrix follows Fisher in applying the concepts of epistemic and aleatory uncertainty, with the former more likely to be a result of human lack of understanding, and the latter more likely to be one of imperfect sources and tools. Truly epistemic uncertainty is the hardest to address, but, as we can see in the taxonomy, there is only one of twelve possible states of uncertainty that is indeed fully epistemic, along with five others that are partially so. Knowing how uncertain a given uncertainty is, makes it much easier to build systems and annotation standards to accommodate it, but it also allows historians and citizen users to maintain their sense of uncertainty itself which is often a complex and potentially hybrid state. Through this instrument, we can see differences between those places that are likely to have wide agreement as to the nature of the uncertainty and what it would take to optimally resolve it (a gap in a primary source), and those areas where the questions are much more individualised, closer to the work of a single historian.

The final taxonomy was deployed as a fundamental element in the PROVIDEDH demonstrator platform for the investigation of uncertainty in historical texts. Its primary functionality was to structure the manner in which users could indicate the nature and level of uncertain passages in the text and to visualise patterns in these annotations. This enabled a like-with-like comparison of annotations across users and documents. The use of the taxonomy in the system (for example in the visual annotator, see Figure 5, is based upon a conceptualisation of uncertainty that has two well-defined sources: 1) tool-induced and 2) human-induced, which can be respectively mapped to the two main categories devised by [Fisher 1999] that were introduced in previous sections of this paper.

The first uncertainty category, tool (or machine)-induced (Figure 5, bottom), can be mapped to aleatoric uncertainty (well-defined objects in Fisher’s taxonomy) and results from the application of computational algorithms (or other historic tools) to the data. These often give their results with a variable degree of bounded uncertainty (e.g., topic models [Blei 2012] or word embeddings [Mikolov et al. 2013]). For this reason, this type of uncertainty is better represented as a continuous probability distribution.

In addition, this representation allows a better understanding of speculative runs of a given algorithm and enhances the what-if analysis process. For example, a researcher could parametrise an algorithm with a fixed set of inputs and launch it a number of times, obtaining a range of mean values and deviations encoded in a document which, if correctly displayed, would allow her to get an idea of how the algorithm behaves. Analogously, the algorithm could be parametrised with a variable set of inputs created by the user running the computation or, as we will see next, by other past researchers. This operation mode would provide an answer to the question “What happens if I run the algorithm a million times using my assumptions?” or “What happens if I run the algorithm a million times using another person’s assumptions?”. As in the case of running the algorithm with the same parameters many times, the results of running the algorithm with different parameters many times could also be summarised in a continuous document allowing the said kind of what-if analysis. This would answer the question of “What happens if I use these other person’s assumptions?” or “What happens if I decide not to trust another actor’s assumptions?”. We argue that the insights obtained from the described experiments are highly valuable for the user, specifically in the case of algorithms that are probabilistic in nature, such as topic models, and whose results – and interpretations – can vary greatly between different runs [Alexander and Gleicher 2016].

The other category, human-induced uncertainty (bottom), arises from 1) direct interpretations of the raw data (which in turn may be based on others’ previous interpretations and grounded expert knowledge of the user) and 2) interpretations of computational analyses performed on the data or 3) a combination of the two. This category is reported by users of our system in a 5-point Likert scale and thus it is best modelled as a discrete probability function. Moreover, we devised the dependency relationships between the categories in our taxonomy to be bidirectional and self-recurring, since, for example, the input parameters and data fed to the algorithms – and therefore their results – are derived from a user’s previous interpretations of textual data and related machine- or human-generated annotations. In turn, these interpretations must necessarily be built upon previous insight obtained by the same or other users who apply computational techniques to the data, effectively forming a temporal belief network [Druzdzel and Simon 1993] that is fixated on the different versions of a dataset. In a first stage, we modelled human-induced uncertainty according to the uncertainty taxonomy of Figure 4 in what was called the 4+1 uncertainty model. Later, and following a series of evaluation sessions and workshops, we evolved the model to support a variable number of human-induced categories. This change was introduced after a series of user-driven experiments and evaluations in which we asked the participants to annotate different types of historical texts (poetry and historical recipes). As we describe in our previous contribution, the uncertainty taxonomy was received positively by the Digital Humanities experts and lay users we engaged with during the design sessions that took part throughout the project [Benito-Santos et al. 2021]. From these experiences, we could check first-hand the latent need, as perceived by the participants, for a more nuanced and flexible way of communicating uncertainty in Digital Humanities projects, as well as the potential benefits that this could bring for collaborative work on Digital Humanities datasets. Specifically, we discovered that, sometimes, users could not pick one of the predefined categories when making annotations on the texts. Furthermore, we also detected that there almost never was a 100% consensus on the uncertainty type different users chose to annotate the same portion of text: e.g., whereas for some a linguistic variant of a word implied imprecision (e.g., they were more or less sure about a word’s meaning, but the word varied from a standard form), for others it meant ignorance (e.g., they did not know what the word meant). These differences, we argue, are often subtle and very much depend on the mental state, ground knowledge on the subject and other traits of the particular user making the annotation. Beyond that, some users showed preference for labelling uncertainty in a particular annotation as a combination of different types. For example, a user could be more or less sure that a word found in the text represents a dialectal variation but at the same time not be sure of the word’s meaning in a certain context. In this case, the user should be able to reflect her views using a combination of categories (e.g., imprecision + ignorance + 0/5 non-credibility + 0/5 gaps). This way of communicating uncertainty, we argue, allows a higher level of expressiveness and is able to capture the nuances of the user’s mental state that needs to be captured and presented to other users working in the same dataset. To allow for even better communication of uncertainty between actors, we allow users to dynamically create new categories as required, which are established on a per project-basis in what is called the n+1 uncertainty model. This departs from the observed effect in our experiments that a group of users working on the same data will naturally establish their own uncertainty taxonomy that covers the needs and particularities of the specific tasks to be performed on the data. As such, they will use it as a communication tool to collaborate in a distributed manner in time and space by appending their annotations to the dataset according to these ad-hoc categories, effectively fulfilling one of the project’s main objectives.

The function of the system ultimately built by the PROVIDEDH team, as a visualisation supported tool for historical research under conditions of uncertainty, was predicated upon the deployment of the taxonomy that underpins it. Interestingly, however, it was also the process of negotiating the taxonomy that allowed the team to function, overcoming the challenges inherent in interdisciplinary work and achieving the goals of their collaborative project. Lee described the effect of “boundary-negotiating artifacts” as twofold, having (1) an effect on the recipient, for whom information is more easily grasped, and 2) an effect on the artifact creator, who must strive to construct an artifact that is easily grasped [Lee 2007]. The results of the PROVIDEDH project can certainly attest to this effect and to the largely invisible, but essential, taxonomy and its development delivered.

This research was funded by the CHIST-ERA programme under the following national grant agreements PCIN-2017-064 (MINECO Spain), 2017/25/Z/ST6/03045 (National Science Centre, Poland), national grant agreement FWF (Project number I 3441-N33, Austrian Science Fund, Austria).

Alejandro Benito-Santos acknowledges support from the postdoctoral grant ’Margarita Salas’, awarded by the Spanish Ministry of Universities.

The authors declared that they have no conflict of interest.