Since the Historische Kranten corpus contains 1,028,555 articles, we calculated with the help of an online tool[22] that a sample of at least 96 articles was needed to reach a 95% confidence level with a 10% confidence interval. This means that with 96 articles, we are 95% certain that our sample is representative of the overall corpus with a deviance of maximum 10%. The confidence interval is actually much smaller (about 5%), since the probability of a word being a location is not 50% but rather 2–3%. We therefore generated a random sample of 100 documents, divided over the three languages proportionally to the overall distribution: 49 French documents, 49 Dutch ones and 2 English ones.[23] The documents range from 1831 to 1970, every decade being covered by at least two documents. We then annotated all mentions of places manually with their positions in the text (first and last character) and manually disambiguated them with their corresponding DBpedia URIs, yielding a total of 662 locations in the following format:

The median number of locations by document is 4.5, ranging from 1 to 62. Most places comprise only one word, but 38 of them contain two and 9 have three words or more. The annotation is partly subjective: one could judge that the correct place is “Yorkshire” instead of “East Yorkshire” for instance, every location having five matching candidates in the dictionary on average. We thus had to validate the list with extra annotators before using it as a GSC.

The Cohen’s kappa coefficient measures inter-rater agreement on a scale between 0 and 1, 0 being zero agreement and 1 total agreement [Cohen 1960]. A value of K greater than .8 is generally considered sufficiently reliable to draw sound conclusions based on the annotation [Carletta 1996]. The kappa is computed as follows: \(\kappa = \frac{\Pr(a) - \Pr(e)}{1 - \Pr(e)}\) where Pr(a) stands for the relative agreement between two raters and Pr(e) for the probability of random agreement.

Our sample of 100 documents contained 30 186 tokens in total, spread over the three languages. For each language, in addition to our own annotation (A), an external annotator (B) was asked for every token to decide whether it was part of a place name or not. Locations containing OCR errors were accepted as long as the annotator could be reasonably sure that it was a place name. Table 2 presents the raw annotation counts, along with the kappa by language.

The average kappa of .91 shows a high agreement that is largely sufficient to consider the GSC reliable. This good score can be explained in part by the relative straightforwardness of the annotation task (LOC versus NON-LOC) compared to more complex ones involving several types of entities, and in part by the detailed instructions provided to the external annotators prior to the task. After some corrections, insertions and deletions, we were left with 654 locations that we mapped to their corresponding DBpedia resources, producing our GSC in the TAC KBP/EDL format, which is slightly different from the one used to annotate the sample:

The impact of OCR quality on entity linking also needed to be evaluated. To do so, we manually corrected the 120 places (out of 654) containing OCR errors and produced a second reference. The original sample and both GSC are also available on the GitHub page of the project.

DBpedia Spotlight[27] allows to find entities in text and link them to DBpedia URIs. Interestingly, the authors pay special attention to quality issues and fitness for use: “DBpedia Spotlight allows users to configure the annotations to their specific needs through the DBpedia Ontology and quality measures such as prominence, topical pertinence, contextual ambiguity and disambiguation confidence”. The four stages of its workflow are spotting, candidate selection, disambiguation, and configuration (emphasis preseverd):

The spotting stage recognizes in a sentence the phrases that may indicate a mention of a DBpedia resource. Candidate selection is subsequently employed to map the spotted phrase to resources that are candidate disambiguations for that phrase. The disambiguation stage, in turn, uses the context around the spotted phrase to decide for the best choice amongst the candidates. The annotation can be customized by users to their specific needs through configuration parameters [ ...].  [Mendes et al. 2011]

However, the original Spotlight was designed for English only, as is the case for many tools. To counterbalance this limitation, Daiber et al. developed a new multilingual version of DBpedia Spotlight, which they claim is faster, more accurate, and easier to configure [Daiber et al. 2013]. This statistical version has been adopted for the online demo.[28] In addition to English, their language-independent model was tested on seven other languages: Danish, French, German, Hungarian, Italian, Russian, and Spanish. The authors reported accuracy scores for the disambiguation task ranging from 68% to 83%.

For the spotting phase, the authors experiment with two methods: a language-independent (data-driven) one based on gazetteers and a language-dependent (rule-based) one relying on more heavy linguistic processing using Apache OpenNLP models.[29] Surprisingly, the language-dependent implementation does not improve the results significantly: it only outperforms the language-independent implementation by less than a percentage point.

The subsequent steps are also fully language-independent: candidate selection is done by computing a score for each spot candidate as a linear combination of features with an automated estimation of the optimal cut-off threshold; disambiguation is performed by using the probabilistic model proposed by Han and Sun [Han and Sun 2011]; finally, configuration allows users to refine the results obtained by setting their own confidence and relevance thresholds, these scores being computed independently of the language.

Developed as a Web content enrichment platform, Zemanta[30] offers a NER API among other services for bloggers. It gained worldwide attention in 2010 when an evaluation campaign showed that it outperformed other state-of-the-art systems for entity disambiguation,[31] an assessment confirmed by later studies [van Hooland et al. 2015] [Hengchen et al. 2015]. Zemanta was subsequently integrated into the NERD framework [Rizzo and Troncy 2012] and into the OpenRefine NER extension.[32] Zemanta requires an API key in order to use its services,[33] but the webpage to apply for one seems to have been down for a long time, preventing new users from registering (although older keys still work). Despite the fact that it officially only supports English text, Zemanta has proved in our own experience to work reasonably well on French and Dutch.

Moro et al. introduce Babelfy,[34] a system bridging entity linking and word sense disambiguation and based on the BabelNet[35] multilingual encyclopaedic dictionary and semantic network which is constructed as a mash-up of Wikipedia and WordNet [Moro et al. 2014]. Aiming to bring together “the best of two worlds”, Babelfy also uses a graph-based approach but relies on semantic signatures to select and disambiguate candidates. The use of these dense subgraphs is very effective to collectively disambiguate entities that would have proven almost impossible to identify separately. Relying on a large-scale multilingual network, Babelfy officially supports 267 languages, in addition to a language-agnostic option.

Precision is consistently ahead of recall, with Zemanta reaching scores between 85% and 90%. The harder task of strong annotation match (taking into account the exact position of each entity in the text) does not affect precision: Babelfy and MERCKX actually improve on their scores, although Spotlight’s precision is cut by a factor of 2. This can be explained by recurrent entities that are correctly identified in all cases. In contrast, all recall scores decrease when considered from the SAM perspective. MERCKX outperforms other systems on recall, but it peaks at 49% (ENT) and 44% (SAM) only.

Low recall scores under 50% can be explained by the multilingual context and by the lack of coverage of DBpedia for some types of locations. Whereas these would be unacceptable in a medical context where failing to retrieve a document can have dramatic consequences, a better precision is generally preferred in less critical applications. MERCKX reaches a F-score just under 60%, a ten-point improvement on both Zemanta and Babelfy which have F-scores under 50%. Spotlight fares disappointingly, with F-scores around the 25% mark. Results on corrected OCR (columns marked “Corr”) will be discussed separately in Section 5.3.3.

MERCKX is heavily dependent on the quality of DBpedia, on which it relies for the disambiguation of entities. The errors of our system can be grouped into three categories, following the typology of Makhoul et al.: insertions, deletions, and substitutions [Makhoul et al. 1999].[37]

Insertions (spurious entities or false acceptances) are entities in the system output that do not align with any entity in the reference. A common factor causing this is multilingual ambiguity. The French adjective “tous”, for instance, when written with a capital “T”, can be incorrectly mapped to the town of dbr:Tous,_Valencia. The type check performed during the construction of the dictionary normally avoids such cases, but some problems can remain when a disambiguation page is missing: in this case, the French resource dbpedia-fr:Tous also points to the Spanish city, with no reference to the adjective. Another frequent mistake occurs when places are mentioned in the name of streets. For instance, the “rue de Lille” in Ypres does not really refer to the French city of Lille, and should therefore not be disambiguated with dbr:Lille. A more elaborate algorithm could try to detect such cases in order to exclude them, but it would be difficult to implement it in a language-independent manner without explicitly blacklisting words such as “rue”, “straat”, “street”, etc.

Deletions (missing entities or false rejections) are entities in the reference that do not align with any entity in the system output. One of the main causes for this is the absence of the dbo:Place RDF type in the resource of a location. For instance, dbr:East_Riding_of_Yorkshire is described as a owl:Thing which is very general and therefore not helpful. However, it is also tagged as a yago:YagoGeoEntity which is more precise. Using multiple types instead of just dbo:Place could improve the recall. Another cause is the absence of a particular label (e.g. when an old spelling is used). The resource dbr:Reims, for instance, does not include a label “Rheims” in any of the three languages used. However, the resource dbr:Rheims does exist and redirects to dbr:Reims. Including redirections in addition to labels could also help to limit the number of missing entities.[38]

Substitutions (incorrect entities) are entities in the system output that do align with entities in the reference but are scored as incorrect. These cases are far more rare than insertions and deletions. Substitutions can be due to the wrong detection of entity boundaries: “Jette” instead of “Jette-Saint-Pierre”, “Flanders” instead of “West-Flanders”. The greedy lookup mechanism of MERCKX normally prevents that, but extra spaces (“West- Flanders”) or long entities (“Jette-Saint-Pierre” contains five tokens because hyphens are tokenized separately) can prove tricky. Another possibility is the attribution of a wrong URI when two places have the same name. No case was detected in our system, but the output of DBpedia Spotlight contains an occurrence of this type of mistake: mapping “Vitry” to “Vitry-le-François” instead of “Vitry-sur-Seine”.

In similar work on Holocaust testimonies, Rodriquez et al. found that “manual correction of OCR output does not significantly improve the performance of named-entity extraction”  [Rodriquez et al. 2012]. In other words, even poorly digitized material with OCR mistakes could be successfully enriched to meet the needs of users. The confirmation of these findings would mean a lot to institutions that lack the funding to perform first-rate OCR on their collections or the manpower to curate them manually.

However, contrary to this study, we see that OCR correction improves the results of all systems. Precision goes up by 1 to 3% on ENT and 5% on SAM in the case of Babelfy. Recall improvement reaches 7% on ENT and over 8% on SAM for Zemanta. Accordingly, F-scores get improved by up to 6% on the corrected version, with MERCKX crossing the 60% mark on both ENT and SAM. This state of affairs can be explained by a number of factors. First, the quality of the OCR seems to be much worse in the case of the Historische Kranten corpus than in the testimonies used for their study: the authors report a word accuracy of 88.6% and a character accuracy of 93.0%, whereas in the case of our sample these scores were somewhat lower: 81.7% (word accuracy on places only) and 85.2% (character accuracy). The overall word accuracy, tested on a subset of the sample, was much lower still: a mere 68.3%. Secondly, the entity linking task is harder than simple named-entity recognition: full disambiguation with an URI is more prone to suffer from OCR mistakes. Using a fuzzy matching algorithm such as the Levenshtein distance could help increase the results without needing manual correction of the OCR. Preliminary experiments with this algorithm indicate that it could lead to an improvement of about 5% F-score, bringing MERCKX close enough to the performance achieved on the corrected version of the sample, although this would come at the expense of efficiency since the Levenshtein distance has an exponential time complexity.