Scholarly workflows in the Digital Humanities can benefit from the application of natural language processing (NLP) tools [Argamon and Olsen 2009]. To serve any practical use to DH scholars, these tools must produce accurate and reliable results for the corpus under study, especially when the intent is to provide linguistic annotations that enable syntactic analyses. For historical texts, such an application is non-trivial because pre-trained tools are usually trained over contemporary materials (e.g., “newswire” articles, microblog text, and general books), making any direct application prone to error [McGillivray et al. 2020]. For trainable language-agnostic pipelines like UDPipe that provide annotations for tokenization, tagging, lemmatization, and dependency parsing simultaneously using Universal Dependencies (UD), it becomes necessary to train UDPipe from scratch using model training data derived from the historical literary materials [Shirai et al. 2020] [Straka and Straková 2017]. Developing these UD annotations can be done through manual effort alone or by applying a pre-trained UDPipe model to the raw text and then later post-editing its output manually for errors [Scannell 2020]. However, the structure of nodes in the dependency tree may also require changing and, for East Asian languages like Chinese and Japanese that can be written without specifying word boundaries other than insertion of punctuation marks, tokenization is often difficult, and thus errors in the tree can be severe when tokenization is inaccurate. Consequently, manual revision can be challenging to carry out in terms of time and expertise needed in working with UD.

Motivated by this issue, the present paper purposes to address the following questions with respect to modern historical Japanese corpora: (1) can accurate UD annotations be developed from scratch using pre-trained tools while also minimizing the amount of manual effort needed for correction, (2) can said generated annotations be used as model training data to achieve improved accuracy on a fundamental NLP task, e.g., word segmentation, (3) does the trained model adapted to historical materials have a substantial effect on the output parsings produced when compared to pre-trained tools, and (4) can the proposed workflow be carried out by non-experts in UD? The answers can be encouraging to DH scholars working with historical corpora who are not subject experts in UD, but would like to make more frequent use of linguistic metadata in their scholarship.

To attempt an answer, this paper introduces a rule-based workflow for modern historical Japanese corpora that produces more accurate UD annotations directly from the raw text in the corpus using the output of pre-trained tools as a starting point. It consists of:

In this way, any parser that learns their model from this data also learns how to deal with historical text.

[Shirai et al. 2020] proposes a related workflow to reduce the amount of manual effort needed for correcting sentence boundaries in modern historical Japanese materials maintained by the National Institute for Japanese Language and Linguistics (NINJAL) [NINJAL 2021]. Their method trains a combination of UDPipe and CRF++ using “core data” manually corrected by experts to predict sentence boundaries on unlabeled data. Our approach also corrects UD annotations, but aims to do so automatically and does not use any “gold-standard” or already corrected data as a starting point.

Text normalization is a well-known preprocessing problem in NLP with many proposed solutions. For Japanese text normalization, [Ikeda et al. 2016] presents a deep learning encoder-decoder model for normalizing Japanese social media and microblog text. Our approach uses text normalization in the context of historical text, which has received recent attention in Bollmann’s survey though Japanese is not covered specifically [Bollman 2019]. Moreover, the goal of our approach is not historical text normalization and involves it only as an intermediary step as part of a larger workflow that aims to provide accurate UD annotations to modern historical Japanese text.

There is a possibility for conflicts to arise during processing. Two main issues are addressed here: (1) overlapping rules, and (2) rows with “blank” forms.

This section evaluates the rule set introduced by this research along three criteria:

1. generalization of rules crafted from the Taiyo corpus to other similar corpora,
2. the effect of the rules on the CoNLL-U output when compared to a parsing without rule application, and
3. evaluating the performance of a word segmentation task when training a NLP pipeline using the proposed rule set.

Indeed, some management of the rule set is needed to achieve an observable improvement. The proposed workflow is not totally automatic and care is needed to ensure rules that are introduced into the collection do in fact lead to more accurate parsings produced by GiNZA (or a pretrained tool of choice) and that overlapping rules do not produce a condition where multiple minima exist. Problems with the former usually present as rows with empty forms that can be detected with ease. Moreover, the amount of manual time needed for review is still reduced as the reviewer need only to concentrate review on the FORM field to obtain improved dependency parsings on historical materials.

While the proposed workflow has an effect on the CoNLL-U output produced by GiNZA and said effects bring a significant improvement in precision and F1 for “B”-label word estimation, the results also point toward a need for developing mechanisms that facilitate expansion of the rule collection that, in turn, furthers the changes made to the dependency structure in the CoNLL-U output; this has the potential to bring more improvement in the performance of trainable NLP pipelines like UDPipe on fundamental NLP tasks for historical materials.

Perhaps one step in this direction are NLP methods that can flag instances of pretrained output with inaccurate parsings that need review, thereby allowing easier rule introduction. Alternatively, another approach is to orient the research towards automatic rule inferencing that pave the path for rapid expansion of the current rule collection. One possibility is to allow for “rule chaining,” that is, the application of one rule that triggers the application of one or more new rules that were not directly applicable on the original sentence. Furthermore, methods from deep learning like the encoder-decoder model shown in [Ikeda et al. 2016] offer rich potential for automatic inferecing. These research directions are appealing, however, caution must be exercised to avoid developing methods that are exciting computationally but offer little to the DH scholars that make use of them [McGillivray et al. 2020]. Therefore, any NLP tool developed for the purpose of aiding analysis with historical materials must first work for the DH corpus at hand.

In this work we introduced a rule-based workflow for providing improved UD annotations to historical Japanese corpora. The principal advantage of our approach is that no “gold standard” data is required for training data development, only the availability of a pre-trained model in the target language. Moreover, the amount of time needed for post-editing pre-trained model output is significantly reduced as the reviewer need only to develop rules that address problems in the FORM field and the review does not require deep expertise in UD. We showed that this “cheaper” review strategy exhibits an effect on the dependency structure in the CoNLL-U output and, furthermore, brings an improvement in the performance of trainable language-agnostic NLP pipelines like UDPipe on word segmentation tasks.

These results are encouraging to DH scholars who would like to enhance their scholarship by annotating historical materials with linguistic metadata that is customizable and more reliable than what would be possible by the straightforward application of “off-the-shelf” tools. Future work will do well to further expedite the manual review needed to achieve good results on the target corpus by incorporating methods that flag pretrained output that is inaccurate and allow automatic rule inferencing. We also caution against the development of techniques that could be interesting for a venue in NLP but offers little for the scholar that would use said techniques on a target DH corpus.

We would like to thank the Department of Computer Science at the University of Miami for providing the computational resources necessary for running the experiments in this research. The work is in part supported by the National Science Foundation Grant CNS P2145800.