During the Middle Ages, Croatian state had undeveloped state institutions, which was mostly caused by weak central government. This was the main reason why Ottoman Empire successfully conquered it in the 15th and 16th century. Therefore, the number of sources from that period is quite limited and they are almost exclusively in Latin.[2]

The test case used for the research was from the middle 19th century census for the old town centre of Zagreb. This census, which was performed in 1857, was the first modern census in Croatia. It is a valuable source of information about the social structure of the 19th century Zagreb. A systematic analysis of the data has yet to be performed due to a number of difficulties. At the time of the census, Croatia was a part of the Habsburg monarchy and the language of the census is German. As the Latin language was dominant in the public service, many Latin name forms were Germanized, and many Czech, Slovak, Italian and German names were Croatized or Latinized. Additionally, there was no standard order of first and last names, many of the first and last names were not separated, and various additional notes were not clearly separated from names, especially when it comes to members of the nobility. Because of all these difficulties, the manual transformation of this source into structured data represents a very tedious and error-prone task.

Our goal was to develop a highly accurate model that can parse name expressions and infer standard names from Zagreb’s middle 19th century census and similar historical texts. There was also a requirement to predict gender and nobility of the person from the name expression. The transcription was available as a text file, and after some standard pre-processing and chunking, 1755 records were extracted, including expressions of person names, dates of births, occupation and similar. We decided to apply standard natural language processing tools to this text.

A common approach to tackle the problem of transforming unstructured images to structured information is designing a natural language programming (NLP) pipeline, wherein the first step includes optical character recognition (OCR)/handwriting recognition (HWR) or transcription. Unfortunately, both of these processes are laden with challenges, and more than often they include a lot of decrypting. The result of converting a historical source to text is usually very messy, and it is commonly referred to as a “dirty” text. As further processing requires relatively clean text, the dirty text usually needs to be corrected – this includes spell-checking, joining separated word parts, etc.

The next phase is tokenisation, i.e. splitting text into tokens, usually words. However, the situation is often quite complicated. Defining what exactly counts as a token (e.g. does it include full stops) is an important decision that can have big impact on further steps. Ready-available tokenizers (usually rule-based) did not perform well on our test case. Therefore, we have applied an aggressive tokenisation in our research, splitting everything into sequences of letters vs. non-letters. Another connected step in the NLP pipeline is the segmentation of text, usually into sentences. Although this step is not exactly applicable to our test set, it is a well-known fact that sentence splitters do not perform well on historical texts, as they are trained on the modern ones [Petran 2012].

The following step would be name-entity recognition (NER), i.e. marking the beginning and end of named entities and classifying each such expression as person, organisation, place, temporal expression, etc. Since serial sources consist mostly of such named entities, successful NER would chunk almost the entire text. Unfortunately, modern NER systems do not perform well on historical text. We have tested some of the best NER systems like Stanford NER, which has an f-score of over 90% on modern texts, but it has performed badly on our test case. We have not the calculated exact measure, but the results were so flawed that it seemed needless to do a formal evaluation.

Another important standard task in this context would be Entity resolution, also called record linkage or record deduplication, where different occurrences of the name of same entities are connected. For example, in parish church books, one person might first be baptised, then married, then mentioned as godfather and finally be deceased. It is worth identifying all these names as referencing the same object, in this case the same person.

Finally, the last phase would be relation or relationship extraction, i.e. extracting relations among entities in the text, for example if a person is born at some place, married to someone, etc. In the broader sense, relation extraction can also include inferring the relationships. For example, if we have a relation being father of between a father and two daughters, we can infer relation being sister between them. Similarly, as the relation being mother of is functional, when it occurs between one entity and two different entities, the system can identify those entities as the same. This is something that is standardly done by semantic web technologies.

Naturally, since the NER systems did not perform well on the test case, it was impossible to apply the existing relationship extraction systems.

It is worth mentioning than an alternative approach to a segmented NLP pipeline could be a technique called joint inference [Poon and Domingos 2007]. In joint inference all levels of processing are performed in the same time, with constraints and information on all levels used for inferring the most probable inference. This approach is more similar to the process of analysis performed by a human researcher.

The segmented parts of the census were available, so our research started with analysing what would a historian working with such data do with them. We first wanted to understand how this process is performed by a human, in order to build an effective computational model. The first obstacle we encountered was understanding proper names. When historians analyse such sources, they often unconsciously extract a lot of information from the name, and these information constrain their search by reducing ambiguity. For example, if two persons in the same part of the text have the same surname, they are probably related; if someone has a female name, it is very improbable that that person is a godfather, etc. Therefore, it is worth extracting these information from the name, especially in the context of joint inference, where all available information should be used to resolve ambiguity.

Parsing of proper names, such as analysing inner structure of names, is not a standard part of NLP pipeline. Parts of text containing proper names are usually recognised by a NER subsystem, and other techniques are used for entity resolution and relation extraction subsystems. The task of parsing proper names does seem to be easy; in the case of personal names, one just has to use regular expressions – first token is first name and second token is last name. After that, the only thing left to do is to check the ending of the first name to classify the person as a female or a male. In reality, situation is much more complex. Even when only the modern names are analysed, there is no easy solution to the name parsing, at least no ready-available system.

To illustrate the problem, even in modern languages there is a great variety of names, name forms and cultural conventions. In the case of famous Icelandic singer Björk Guðmundsdóttir, her second name is not really a surname and one should address her with full name. This means that you would find her listed in address book under letter B not G. Similar case is with Arabic, Chinese, Russian, Polish, Serbian and other names that usually start with the last name. Spanish people traditionally have four names, and are addressed by the third one, and the situation in even more complicated in Brazilian Spanish.

The situation with names is even more complex in historical contexts. Until the second half of the 18th century, which is relatively recent, there were no standard name forms in many European countries. In the example of the first Croatian king, Kralj Tomislav, “Kralj” means king – it is a title, not a first name, and “Tomislav” is the first name. In the modern language, however, “Kralj” is a quite common last name. The situation gets even more complicated with noble titles, maiden names, etc.

It would be very helpful to have an accurate proper name parser, the results of which could be used in a more advanced analysis. To the best of the authors’ knowledge, this problem did not receive much attention in NLP related communities.

Parsing name records into constituent parts can be modelled as a sequence labelling problem, viewed as a special case of part-of-speech tagging and shallow parsing. Although this particular problem has not received much attention in recent literature, extensive work has been done on the related and more general problems of Part-Of-Speech (POS) tagging, shallow parsing and named entity recognition [Graves 2012] [Osborne 2000] [Ndeau and Sekine 2007].

Early sequence labelling systems were rule-based; for example, those developed by Brill [Brill 1992], which are still used in some application domains [Chiticariu 2010]. Today, the best performing models are probabilistic, and are generally based on probabilistic graphical models. In particular, models that use conditional random fields represented the state-of-the-art models [Sha and Pereira 2003] [Viet Cuong at al 2014], until the recent usage of deep learning models [Devlin at al 2018].

However, studies on the application of probabilistic models on historical texts have yet to yield satisfactory results. It is very tedious to annotate historical texts, especially when many different sources have to be analysed and the reuse of existing training datasets is not a feasible option. Another reason why there is a need to consider alternatives to the probabilistic approach is that, due to the noisiness of historical sources, the integration of sequence labelling with a joint inference model [Sha and Pereira 2003] is promising as an alternative to the use of a traditional language processing pipeline. Joint inference can reduce OCR ambiguities, and an approach that combines text correction and sequence labelling with the higher-level syntax, semantics and historiographical constraints is more representative of the way in which a human historiographer would perform the task. For example, the probability of social status depends on location, members of the household and place of origin, and ambiguous last names can be resolved by family member records. Although joint inference is compatible with the probabilistic approach [McCallum 2009], the rule-based approach seems more promising in this domain. As the Markov Logic Networks, a framework that Sha and Pereira [Sha and Pereira 2003] used, does not readily scale to this kind of problem, we selected a rule-based framework that was based on stable model semantics.

Stable model semantics can be viewed as the semantics of logic programming. So the rules can use (almost) the full expressivity power of the first-order logic. A further benefit relates to the non-monotonicity of negation as failure, which enables easy modelling of interaction among general and specific rules. Answer set programming (ASP) implements stable model semantics. Due to modern, highly optimized grounders and SAT solvers, ASP implementations are fast enough for many applications and are mostly used for high-level reasoning tasks such as planning, diagnostics, learning and scheduling [Gelfond 2008]. This framework also looks very promising for NLP applications [Balduccini 2013], and especially for our problem. The leading ASP modelling language, Potassco, which was developed at the University of Potsdam [Gebser at al 2011 ], includes support for weak constraint and optimization. This enables the formalization of the sequence labelling task as an optimization problem and, therefore, seems particularly promising in the context of the joint inference model.

The test set for the models consisted of 1774 transcribed records from the census, including name expressions, gender and nobility labels. The available dataset of 4018 labelled modern international names (12,075 tokens) was used for the initial training and test dataset.

The pseudo code for the preliminary rule induction system was as follows.

The hyper-parameters maxLevel and maxCardinality control the number of level of specific rules and the number of atoms in rules. For performance reasons, the learning system was implemented in Python and Potassco ASP. Trained on the modern language training set, the system generated 218 rules on four levels of generality. Token level f1-score, measured on 20% of the modern data-set was 0.95.

The gender and nobility prediction model was based on character n-Gram (length 1-9) of name expressions in census data. Name expressions were pre-processed by marking the beginning and end, lowercasing and stripping accents. A support vector classifier with linear kernel was used, and the parameters were obtained by grid search. As the results of k-folding cross validation were satisfactory, the initial plan for building rule-based classifiers was abandoned.