The concepts embedded in humanities materials are unstructured and possess multifaceted attributes. New insights from these materials are derived through a systematic qualitative study. However, for the purpose of quantitative analysis using digital humanities (DH) methods and tools, it becomes imperative to formalize these concepts. The functionality of DH is contingent upon the deployment of formalized concepts and models. Formalization converts unstructured data into a more structured form and is applied to various fields such as computer science, medicine, and mathematics. Concurrently, models serve as representations created to closely examine the modeled subject, while metamodels define the structure and properties of these models. Fundamentally, metamodels provide a framework for the analysis and construction of models that are applicable to and beneficial for a predetermined class of problems [Tomasi 2018, 173]. Formalization and models constitute essential elements in both applied and theoretical DH. Michael Piotrowski defines applied DH as concerned with constructing formal models for phenomena studied by their “mother disciplines” (e.g., digital history and digital literary studies) and their methodology [Piotrowski 2022a, 3]. Theoretical DH, on the other hand, studies the general properties of formal models in the humanities at a higher level of abstraction. The objectives of these sub-fields involve theoretical DH creating and studying metamodels, while applied DH apply these metamodels to research questions [Piotrowski 2022a, 3]. Documentations outlining the advantages, disadvantages, challenges, and constraints of algorithms are crucial for constructing metamodels.

Piotrowski discusses formalization and models in the humanities and DH. He contends, based on the works of Gilles-Gaston Granger, that formalization in humanities not only extracts structure from the unstructured through hermeneutic interpretation but also invents “new structures”  [Piotrowski 2022a, 11]. Similarly, some disciplines, such as linguistics in humanities, have been using models to study grammar [McCarty 2004]. On the contrary, formalization and modeling in DH are used to computationally operationalize “scholarly arguments”  [Ciula et al. 2018, 347]. In order to derive “new structures”  [Piotrowski 2022a, 11] especially for domain-specific humanities materials using computational study, it is imperative to formalize and model the materials. The absence of formalized concepts and models for studying the British India colonial corpus, in this regard, hampers the computational application to address humanities research questions quantitatively.

Since “mapping and surveying have been seen as instrumental in extending” the colonizers control over land to establish their government [Ehrlich 2023, 193]. This resulted in transcending the survey from military and geographical to granule details of “natural history, political economy, and every conceivable species of inquiry into native society”  [Ehrlich 2023, 196]. The documents hence are intricate, and the format is rather non-standard as they are “chaotic, if not anarchic, character”  [Edney 1997, 162]. In this scenario, the British India colonial corpus cannot be effectively utilized for topic-specific research questions employing advanced text mining without formalizing the concepts within it. For instance, in the corpus curated to study water-related features in the Madras Presidency[1] in British colonial South India using DH methods, data about water is not explicitly provided in most texts unless they are specifically dedicated to hydraulics. Subsequently, extracting information requires strenuous close reading due to the presence of heterogeneous concepts in these texts.

For instance, Madras District Manual Coimbatore Volume II (1898) contains diverse data about population, religion, caste, marriage, and many other details, along with information about water features in Coimbatore. While it serves as a comprehensive manual for the Coimbatore district, the challenge lies in extracting specific water-related data from this text. Disregarding other potentially relevant data is not feasible, as it may have connections with water-related information. Identifying such connections or potential concepts for topic-driven questions poses a challenge. Apart from this text, several others, such as Handbook Of The Madras Presidency (1879), Madras District Gazetteers Coimbatore District (1880), and Report On The Administration Of The Madras Presidency During The Year 1869–70 (1870), encompass a variety of concepts, including attributes relevant to water. During the curation of these texts, keyword searches were employed, using terms like Coimbatore, water, canal and river etc. While this method revealed information related to the specified keywords, relying solely on a corpus curated through simple keyword searches presents challenges. As pointed out by Oberbichler and Pfanzelter, the complexity of language “characterized by ambiguity and concepts that are difficult, if not impossible, to trace by computational methods and thus keyword searches alone”  [Oberbichler et al. 2021, ¶18]. Therefore, formalizing the multifaceted concepts within the texts is imperative for further computational study. In this article, the questions I ask are as follows: What constitutes the optimal approach for identifying multifaceted concepts within the non-standard British colonial India corpus through models, and how can these concepts be formalized using formal models? How can metamodels be developed based on this experiment for a similar corpus? The aim of this article is to propose proficient concept-based models to formalize heterogeneous concepts from the non-standard British colonial India corpus using computational models, and subsequently, to establish metamodels through experimental methods for the construction and analysis of analogous corpora.

This article is divided into six sections following this introduction. Section 2 focuses on the conceptual theoretical framework for constructing computational models to formalize concepts. It elucidates how models can be constructed using the infrastructure of the texts through concept diagrams and illustrates the model through a theoretical example. Section 3 outlines the methodology of the experiment, encompassing preprocessing, the application of computational models, and the visualization of the extracted models. Section 4 examines the extracted models by illustrating selected texts from the corpus to assess the efficacy of the proposed models. It discusses two kinds of outcomes from the experiment: one delineates the distribution of the concepts both in sub-models and primary models and the other explores the features of the models. This section also includes the disadvantages and challenges of the experiment. Section 5 delves into the metamodels using concept diagrams to develop a similar domain-specific non-standard corpus. Section 6 presents the conclusion of this article.

Corpus building is an integral part of the machine learning process, especially when dealing with a large amount of text. Oberbichler and Pfanzelter discuss the rewarding nature of identifying patterns for research questions in big data using quantitative methods; however, it oftentimes comes with its own intriguing challenges “to find and extract those parts in the massive data dumps that are relevant for the topic in question”  [Oberbichler et al. 2021, ¶6]. As they say, identifying conceptual patterns within the British India colonial corpus is notably challenging owing to its intricate content infrastructure. This infrastructure exhibits variations across texts and incorporates a multitude of concepts and topics which hamper the application of formal models. Given that my research focuses on extracting information about water features in the corpus using computational methods, my inquiry revolves around understanding the discourses related to water and water-related features. Specifically, I aim to investigate the transformation of water bodies in the Coimbatore region. The goal of constructing this corpus is to present pertinent headings, sections of texts, and complete texts that address my research question. This will facilitate further quantitative and qualitative analysis.

Relevant methodologies for constructing and analyzing corpora based on concepts, topics, indices, and representativeness can be found in DH literature [Jähnichen 2017] [Englmeier et al. 2021] [Oberbichler et al. 2021] [Verheul et al. 2022]. Nevertheless, building and applying formal models to examine British India colonial corpus has never garnered attention within the realm of DH. The concepts embedded in this historical corpus not only hold significance for comprehending the past but also aid in establishing connections between the present and past. The proposed methodology is designed to extract both primary and secondary heterogeneous concepts. While the latter may not always be the primary focus of British India colonial manuscripts, it remains relevant for topic-oriented research.

Discussions concerning models are pivotal in the field of DH [McCarty 2004] [McCarty 2005] [Buzzetti 2002] [Beynon et al. 2006] [Flanders et al. 2015] [Ciula et al. 2018]. William McCarty defines a model as “a representation of something for purposes of study or a design for realizing something new”  [McCarty 2004]. This definition draws upon Clifford Geertz’s analytical differentiation between a model (representation) of something, exemplified by a grammar describing the features of a language, and a model (design) for something, akin to an architectural plan providing design [McCarty 2004]. Typically, models inherently exhibit simplification and shed light on previously unknown aspects. However, models within the humanities often maintain only partial explicitness, commonly expressed informally through natural language [McCarty 2004] [Piotrowski 2019]. In contrast, the natural and engineering sciences extensively employ explicit and formal mathematical models [Epstein 2008] [Piotrowski 2019]. In the realm of computing, as Brian Smith, cited by McCarty, emphasizes, formal “models are fundamental,” running by manipulating representations, always formulated in terms of models [McCarty 2004]. Concerning formal models, the term “formal,” as elucidated by Piotrowski, signifies being “logically coherent + unambiguous + explicit” and in the domain of DH, a requisite degree of formalization is crucial to enable models to be processed and manipulated by computers, referred to as computational models [Piotrowski 2019]. Additionally, a metamodel, employing graphical representation and natural language to elucidate the criteria and parameters essential for the computational process, can inform computational model to improve the efficiency of the process. These metamodels are methodically transformed into computable implementations through varying levels of formality in modeling [Ciula et al. 2018]. While metamodeling remains relatively unexplored in DH, modeling, in general, is regarded as one of the fundamental research practices in DH. Nevertheless, the goal of a computational/formal model in DH is to mitigate complexity and uncertainty through the logical structure of the model.

In similar to the definition of model, the definition of a concept varies across disciplines, with philosophers viewing it as an inquiry into changes manifested through various events, linguists identifying concepts in onomasiological words and mapping conceptual changes over different periods, and historians exploring various concepts and their significant changes throughout periods in corpora using computational models [Brigandt 2010] [Linguistic DNA] [Verheul et al. 2022]. In this context, the concept, as defined for this specific study, can be inferred from the organization of words and their contextual relationships. Nonetheless, to identify, trace and inquire concepts, first understanding and identifying them in the corpus is imperative. In this case, the formal model plays a crucial role in formalizing and extracting concepts through its logical framework which interconnects the former and the latter. The concept-based formal model endeavors to capture meaningful concepts by processing raw data, facilitating organization, correlation, and the establishment of a semantic network among concepts. This, in turn, enables more nuanced and context-aware analysis or decision-making. In this case, my goal is to formalize and extract the heterogeneous concepts hidden and layered in the British India colonial manuscripts using computational models.

To formalize the concepts, given the non-standard format of the corpus, the models can be divided into three kinds: sub-model, primary model, and larger model, as the concepts are layered at various levels in the texts, as demonstrated in the concept diagrams in Figures 1 and 2. The concepts embedded in the content of each heading are classified as sub-models and those present in the entire text are labeled as primary models (see Figure 1). The interconnection between these models can be used to develop the larger model, as the similarity network drawn in the diagram (see Figure 1). For a more nuanced comprehension of the model, let us study a hypothetical elucidation based on my research about the water in the colonial period. In Figure 2, number one enclosed in a red circle signifies the interconnection between the primary model “rivers and channels” and a sub-model “waterbodies and canal company” derived from text 3 and 1; number two delineates the correlation between a sub-model “water scarcity” and a sub-model “water infrastructure” identified in text 3 and 1; number three establishes a connection between a sub-model “public and investment” and and sub-model “dam construction” found in text 1 and 3; number 4 links a sub-model “water tax” and the primary model “waterbodies and canal company” observed in text 2 and 1, while number 5 designates the non-connected primary model of text 2 with any of the models.

During the British colonial period, the concept of water extended from the primary resource for drinking, farming, and ritual entity to a taxable resource and expandable colonial revenue. Such an attitude has resulted in focusing on “developing irrigation infrastructure through building larger dams” [Saravanan 2020, 37]. Ratna Reddy says “[t]he interest of the British Government in increasing public investment in irrigation was to gain higher revenue”  [Reddy 1990, 1047]. The colonial government approved a private entity, the Madras Canal Company, to invest in irrigation projects. This Company started to focus on building bigger water bodies and dams. The British colonial government “used all possible ways to extract maximum revenue in the name of irrigation development”  [Reddy 1990, 1057].

In such scenarios, the aforementioned hypothetical concepts elucidate concerns regarding water during the colonial period, excluding the primary model “political and movement” from text 2. Despite the relevance of the sub-model “water tax” from text 2 to other models, the logical framework of the formal model for the primary model might overlook the sub-model “water tax.” This oversight is due to the primary model being designed to extract the most crucial concepts from the entire text, and the aforementioned sub-model may not be estimated a pivotal concept. However, it remains pertinent to topic-driven research. For further analysis, one can employ additional machine learning models for a quantitative study of the selected sub-models and primary models or curate these models for a qualitative study. Alternatively, the corpus can undergo reorganization by filtering texts, informed by these models. In every aspect, these models contribute to our exploration, categorization, and understanding of the corpus.

For this conceptual and theoretical framework, I chose an unsupervised approach using Affinity Propagation Algorithm (APA). The APA is widely used in many diverse fields for data mining such as medicine, chemistry and bioinformatics etc. Brendan J. Frey and Delbert Dueck introduced APA in 2007 [Frey et al. 2007]. It has a number of advantages compared to other clustering algorithms such as Kmeans. Kmeans requires a number of clusters to group words (or in other words data points) gleaned from its similarities. On the other hand, APA does not entail the number of clusters and it takes “input measures of similarity between pairs of data points and [r]eal-valued messages are exchanged between data points until a high-quality set of exemplars and corresponding clusters gradually emerges”  [Frey et al. 2007, 972]. If the data point i is similar (s) to the data point k which is determined through negative Euclidean distance, then s(i, k) is s(i,k) = −||xi − xk||2 [Frey et al. 2007, 972]. and Levenshtein Distance is employed to calculate the distance between data points. It measures the distance between the data points based on their similarity and assigns a weightage to each word.

The communication between data points (i, k) happens through responsibility(r) and availability (a). The r on behalf of data point i will send a message to check the a of the k and looking for a s pattern. If k does not have similarity, then i will look for other data points. It is an iterative process till the data point i finds its group and vice versa [Frey et al. 2007, 972]. The other advantage feature of APA is “exemplar” which will be chosen as the “representative” of the cluster. For instance, if the data points (k, k) have larger values than other data points in the same cluster, it will be selected to represent the cluster. APA also chooses the number of clusters according to the density. This algorithm is applicable to the corpus like British India colonial as we cannot presume the number of clusters as the number could vary from text to text in the corpus. APA is efficient when it comes to small amounts of data. I applied APA to contents in each heading of text in the corpus for extracting sub models and also harnessed to each text separately for mining the primary models, but not to the whole corpus.

The corpus that I use for this article is part of an AHRC funded project “Digital Innovation in Water Scarcity” at Lancaster University in which I worked as a Research Associate. The primary aim of this project is to study the water transformation from early ninetieth century British India Coimbatore to present through diverse corpus of historical maps, texts and oral testimonies through various innovative DH methods. The British colonial India corpus is curated from various open access online platforms such as Internet Archive, Google books and National Library of Calcutta. These texts are out of copyright. This corpus is small and consists of 29 texts (see Figure 3). The corpus is cleaned as they were curated as PDFs and image files through third party with help of project fund.

In the preprocessing, each text in the corpus is divided into a number of parts such as titles, table of contents, preface, index and body of the content. These parts are marked using hyperlinks tags. These tags are used to extract table of contents and body of the content. The tagged “Table of contents” and “Body of the content” are cleaned to remove the punctions and other symbols as they encumber mining the clusters. However, this preprocessing has a few challenges since not all historical texts are standardized using the table of contents. Three texts did not have table of contents but they have headings inside the text and five texts did not have neither table of contents nor headings inside the text. Since all headings inside texts are in uppercase and most of the table of contents are also in uppercase. The headings in uppercase facilitated extracting their content using first heading as start and second head as end. However, the last heading, of course, would not have a subsequent heading for which I added “End of the Doc” at the end of the text. Then I extracted the contents inside the headings for twenty-four texts using regular expressions. For the remaining five texts, I had to extract the body of the text as there is no option to separate the content using table of contents. I used loop function to run the text one by one in Python. After preprocessing, I converted the extracted data into DataFrame using Pandas which later passed to APA. Kmeans[2] algorithm in Sklearn is applied to predict the clusters through elbow method, which is applied to “test the consistency of the best number of clusters by comparing the difference of the sum of square error,” to compare the efficiency of means and APA to improve the efficiency [Umargono et al. 2020, 121]. The proficiency of the latter is much better than the former. I also decided to extract top fifteen words in each cluster after a few trials. Finally, the data in dataframe, prediction and the top features of the cluster passed into the APA’s similarity function to extract clusters and exemplars for the content in each heading and the entire text and both are stored into separate CSV files. Then I created three kinds of visualizations using Tableau and RAW online graph: representing the total number of clusters across the corpus; distinguishing between the sub-model and primary model through selected two texts and visualizing the models from the selected text for further investigation.

In the realm of computer science and related fields, a metamodel has surfaced, serving the purpose of “facilitating conceptual modeling, defining constructs of conceptual modeling languages, specifying constraints on the use of constructs, and encoding the similarities of different models”  [Jeusfeld 2009, 1728]. Jeusfeld characterizes a metamodel as encompassing several models that include models, parameters, features, challenges, and limitations. Metamodels are designed “to build explicit” models, which are meticulously delineated to enable a thorough understanding of their contents [Epstein 2008]. Their significance in theoretical DH lies in facilitating a computational approach to studies in the humanities. Piotrowski underscores the importance of metamodels in theoretical DH, asserting that “the theoretical digital humanities create and study the metamodels whose concrete application to research questions in the disciplines of the humanities is the object of the applied DH”  [Piotrowski 2022a, 3].

In the absence of an existing metamodel for the non-standard corpus to formalize the concepts within theoretical DH, I have formulated one based on processes, results, experiments, and experiences and represented it through concept diagrams. This metamodel is poised to significantly contribute to similar corpus-building research, serving as a valuable resource to improve future endeavors by addressing errors noted during experimentation and redesigning for enhanced comprehension.

Model 1 provides a detailed depiction of the corpus selection and pre-processing (see Figure 8). Establishing a corpus involves the construction of a model. Consequently, modelers must address inquiries such as: What constitutes the original? In what ways does the model serve as a reduction of it? And for whom and for what purpose is the model being created? [Piotrowski 2022b, 90]. Curating texts for a corpus pose significant challenges, including considerations about the collection of texts, copyright status, and existing machine-readable formats for materials. Despite this, the curated corpus is primarily guided by a general understanding of the research field. Consequently, eliminating the most and/or less relevant materials from the corpus is neither straightforward nor transparent. Subsequently, preprocessing becomes a crucial step in Model 1, marking the initial phase of any quantitative study. The texts should not only be in machine-readable formats (such as plain texts or PDF files) but also formalized using Text Encoding Initiative or hyperlink tags, particularly essential for non-standard corpus. A notable challenge in digital humanities methods for historical corpus is the presence of spelling variations in the text [Gregory 2014]. Acknowledging the impact of spelling issues on the models is paramount. Being cognizant about the influence of spelling issues in the models is pivotal. The nature of the research inquiry will determine whether the texts in the corpus necessitate deep cleaning, involving not only the removal of stop words but also stemming and lemmatization of words, significantly influencing the results in the subsequent model.

Model 2 delineates the parameters and criteria governing the construction of the corpus through concept-based models (see Figure 9). The determination of whether to ground sub-models and/or primary models is contingent upon event- or topic-based research inquiries. Additionally, the choice between a single algorithm or a combination of algorithms for extracting concepts and their models significantly influences the selection of relevant texts for the corpus. The decision is also influenced by the nature of the research, as mining sub-models may be appropriate for some topic-driven research, while extracting both sub-models and primary models may be essential for others. In cases where the corpus is intricate and comprises various forms and genres, such as documents, surveys, and reports, building the corpus based on both sub-models and primary models is deemed more fruitful. However, the selection of appropriate algorithms is crucial, as it determines the outcome of the quantitative study.

Moreover, the process of choosing models to represent concepts is pivotal and can be achieved either manually or through an algorithm. For example, if one opts for APA due to its exemplar feature, which is significant for the corpus used in this article, relying on a single exemplar from the cluster may sometimes be misleading. This bias arises as the concept is determined based on the preferred representative, leading to potential distortion. To mitigate this bias, selecting multiple exemplars based on the density of the cluster will, to a certain extent, evade this issue in concept-based models. However, this model presents other potential challenges, such as running headings on each page and an increase in the number of outliers due to fewer words in the content of the headings. Setting parameters to deriving concepts and models for headings with only a few sentences can address this issue; however, it may also result in the neglect of crucial data. Decisions regarding such trade-offs can be made through a process of trial and error.

Model 3 outlines the parameters for developing a larger model for the corpus (see Figure 10). Larger models can be generated from three types of datasets: sub-model, primary model, and standalone model. It can be created through a identifying the semantic network between sub-models and primary models. Such semantic network can also be employed to reorgainze the texts in the corpus based on the priority and to develop a sub-corpus. Comparing the three models would be worthwhile to explore the dispersal of concepts and their models throughout the corpus.

The British colonial India corpus, which has not received attention in the field of DH, poses considerable challenges and drawbacks due to its intricate format and diverse concepts. This research seeks to confront these challenges by introducing concept-based formal models designed to formalize the heterogeneous concepts present in corpus, leveraging computational methods. In summary, this study illustrates the potential and challenges inherent in the development and application of formal models for the specified corpus. Derived from the investigation, several promising findings emerge:

1. Quantitative studies in DH typically prioritize the identification of trend patterns within a corpus. However, the computational approach proposed in this work assigns equal importance to both trend and non-trend patterns in the text, as the latter also significantly contributes to shaping the overall trend pattern. This stands in contrast to traditional approaches to studying historical concepts, which often rely on limited texts through qualitative studies or use computational methods to identify broader conceptual patterns. The recommended model enables a comprehensive study of concepts, even within extensive datasets.
2. Additionally, the concept-based model emerges as a convincing and promising framework, not only illustrating how concepts are distributed across the text and its headings but also capable of formalizing nuanced concepts to construct a more comprehensive model. The construction of the concept involves studying exemplars and their associated terms, underscoring the significance of domain knowledge in the decision-making process. Nevertheless, this model introduces a novel computational approach to trace concepts and provides insight into the curation of colonial knowledge.
3. The application of the theoretical framework, as outlined in Section 2, is highly effective in exploring the chosen corpus. This model proposes to categorize the text into two types: the text itself and the units of the text based on its peritext. The development of formal models can be accomplished using this categorization. However, as mentioned in Section 3, some challenges arose in these divisions, as a few texts could not be segmented into units due to the absence of peritext. Nevertheless, these challenges were addressed by manipulating the formal models to extract units based on categorization within the text (headings).
4. The analysis in Section 4 reveals that the number of models between sub-models and primary models may vary based on the distribution of concepts within the unit and the text. The examination of the extracted models from the chosen text underscores the significance of the sub-model in deriving nuanced concepts from the text, which are overlooked in the primary model and vice versa. However, in the process of constructing and studying the corpus for topic-specific research, both models are indispensable and can mutually enhance exploration of the concepts within the text, aligning with the primary objective of this proposed model.
5. Lastly, Section 5 delineates the metamodels, a facet not prioritized in theoretical DH. Derived from the experiment, it provides insights into the general properties, possibilities, and challenges involved in studying a similar corpus.

The findings of this article propose several directions for future research:

1. The current corpus is relatively modest in size. In subsequent work, an expansion of the corpus by curating additional texts is planned and the texts will be categorized based on its forms and genres, including documents, surveys, and reports. Moreover, the adoption of TEI guidelines to generate tags for the texts will be considered. Systematic categorization and tagging systems aim to enhance the formalization process.
2. While the APA computational algorithm demonstrates efficacy for the proposed model, there are limitations in the exemplar features and clustering patterns, notably a surge in outliers, and not all exemplars convey meaningful insights. Future investigations will explore alternative advanced computational models or combinations thereof to achieve more efficient pattern clustering. Additionally, efforts will be made to fine-tune parameters for extracting more than one exemplar per cluster based on content density.
3. The next research focus involves formalizing the spelling of Indian names. As highlighted in the final subsection of Section 4, a considerable number of Indian names did not appear in the cluster due to inconsistencies in their spellings. Ongoing efforts are dedicated to addressing this issue, with a specific emphasis on formalizing concepts based on Indian names [Shanmugapriya 2023].

I express my gratitude to Mohanapriya, the software trainer, for her assistance in preparing the codes for the entire project. Many thanks to Bhavani Raman, a historian at the University of Toronto Scarborough, for providing valuable insights into the format and non-standard nature of the British colonial India corpus. I am also thankful to the AHRC for funding the “Digital Innovation in Water Scarcity Coimbatore India” project, through which the corpus was collected and cleaned using project funds. Additionally, I appreciate Deborah Sutton, who served as the Principal Investigator of this project at Lancaster University.